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FOREWORD 


These  proceedings  contain  the  papers  of  the  International  Conference  e-Leaming  2013,  which 
was  organised  by  the  International  Association  for  Development  of  the  Information  Society  and 
co-organised  by  The  University  of  Economics  in  Prague  (VSE),  Czech  Republic,  23  - 26  July, 
2013.  This  conference  is  part  of  the  Multi  Conference  on  Computer  Science  and  Information 
Systems,  22  - 26  July  2013,  which  had  a total  of  948  submissions. 

The  e-Eeaming  2013  conference  aims  to  address  the  main  issues  of  concern  within  e-Eeaming. 
This  conference  covers  both  technical  as  well  as  the  non-technical  aspects  of  e-Eeaming. 

The  conference  accepts  submissions  in  the  following  seven  main  areas:  Organisational  Strategy 
and  Management  Issues;  Technological  Issues;  e-Eeaming  Curriculum  Development  Issues; 
Instmctional  Design  Issues;  e-Eeaming  Delivery  Issues;  e-Eeaming  Research  Methods  and 
Approaches;  e-Skills  and  Information  Eiteracy  for  Eeaming. 

The  above  referred  main  submission  areas  are  detailed: 

->  Organisational  Strategy  and  Management  Issues 

- Higher  and  Further  Education 

- Primary  and  Secondary  Education 

- Workplace  Eeaming 

- Vocational  Training 

- Home  Schooling 

- Distance  Eeaming 

- Blended  Eeaming 

- Change  Management 

- Educational  Management 

- Continuous  Professional  Development  (CPD)  for  Educational  and  Training  Staff 

- Return  on  e-Eeaming  Investments  (ROI) 

Technological  Issues 

- Eeaming  Management  Systems  (EMS) 

- Managed  Eeaming  Environments  (MEEs) 

- Virtual  Eeaming  Environments  (VEEs) 

- Computer-Mediated  Communication  (CMC)  Tools 

- Social  Support  Software 

- Architecture  of  Educational  Information  Systems  Inffastmcture 

- Security  and  Data  Protection 

- Eeaming  Objects 

- XME  Schemas  and  the  Semantic  Web 

- Web  2.0  Applications 


XI 


e-Learning  Curriculum  Development  Issues 

- Philosophies  and  Epistemologies  for  e-leaming 

- Learning  Theories  and  Approaches  for  e-leaming 

- e-Leaming  Models 

- Conceptual  Representations 

- Pedagogical  Models 

- e-Leaming  Pedagogical  Strategies 

- e-Leaming  Tactics 

- Developing  e-Leaming  for  Specific  Subject  Domains 

Instructional  Design  Issues 

- Designing  e-Leaming  Settings 

- Developing  e-Leaming  Pilots  and  Prototypes 

- Creating  e-Leaming  Courses 

- Collaborative  learning 

- Problem-based  learning 

- Inquiry-based  learning 

- Blended  learning 

- Distance  learning 

- Designing  e-Leaming  Tasks 

- E-leaming  activities 

- Online  Groupwork 

- Experiential  learning 

- Simulations  and  Modelling 

- Gaming  and  edutainment 

- Creativity  and  design  activities 

- Exploratory  programming 

e-Learnlng  Delivery  Issues 

- e-Delivery  in  different  contexts 

- Higher  and  Further  Education 

- Primary  and  Secondary  Schools 

- Workplace  Learning 

- Vocational  Training 

- Distance  Learning 

- Online  Assessment 

- Innovations  in  e-Assessment 

- e-Moderating 

- e-Tutoring 

- e-Facilitating 

- Leadership  in  e-Leaming  Delivery 

- Networked  Information  and  Communication  Literacy  Skills 

- Participation  and  Motivation  in  e-Leaming 

e-Learnlng  Research  Methods  and  Approaches 

- Action  Research 

- Design  Research 

- Course  and  Programme  Evaluations 

- Systematic  Literature  Reviews 


- Historical  Analysis 

- Case  Studies 

- Meta-analysis  of  Case  Studies 

- Effectiveness  and  Impact  Studies 

- Evaluation  of  e-Eeaming  Technologies 

- Evaluation  of  Student  and  Tutor  Satisfaction 

- Teaming  and  cognitive  styles 

- Ethical  Issues  in  e-leaming 

e-Skills  and  Information  Literacy  for  Learning 

- Teaching  information  literacy 

- Electronic  library  and  information  search  skills 

- ICT  skills  education 

- in  schools  and  colleges 

- for  business,  industry  and  the  public  sector 

- in  adult,  community,  home  and  prison  education 

- informal  methods  (peer  groups,  family) 

- Education  for  computer-mediated  communication  skills 

- Netiquette 

- Online  safety  for  children  and  vulnerable  users 

- Cybercrime  awareness  and  personal  prevention 

- Student  production  of  online  media 

- Web  design 

- Digital  storytelling 

- Web  2.0  tools 

- etc. 

- Digital  media  studies 


The  e-Eeaming  2013  conference  received  231  submissions  from  more  than  36  countries.  Each 
submission  has  been  anonymously  reviewed  by  an  average  of  four  independent  reviewers,  to 
ensure  that  accepted  submissions  were  of  a high  standard.  Consequently  only  36  full  papers 
were  approved  which  means  an  acceptance  rate  of  16  %.  A few  more  papers  were  accepted  as 
short  papers,  reflection  papers  and  posters.  An  extended  version  of  the  best  papers  will  be 
selected  for  publishing  as  extended  versions  in  the  Interactive  Technology  and  Smart  Education 
(ITSE)  journal  (ISSN:  1741-5659)  and  also  in  the  lADIS  International  Journal  on 
WWW/Intemet  (ISSN:  1645-7641).  Other  outlets  may  also  receive  extended  versions  of  the 
best  papers,  including  journals  from  Inderscience 

Besides  the  presentation  of  full  papers,  short  papers,  reflection  papers  and  posters,  the 
conference  also  included  two  keynote  presentations  from  internationally  distinguished 
researchers.  We  would  therefore  like  to  express  our  gratitude  to  Thomas  C.  Reeves,  Professor 
Emeritus  of  Teaming,  Design,  and  Technology,  College  of  Education,  The  University  of 
Georgia,  USA,  and  also  to  Doc.  Ing  Jan  Eojda,  President  of  the  Czech  Association  of  Distance 
Teaching  Universities,  Czech  Republic,  for  accepting  our  invitation  as  keynote  speakers.  Also  a 
special  thanks  to  Peter  E.  Stanchev,  Kettering  University,  USA  and  Institute  of  Mathematics 
and  Computer  Science,  Bulgarian  Academy  of  Sciences,  Bulgaria,  for  presenting  a tutorial. 


A successful  conference  requires  the  effort  of  many  individuals.  We  would  like  to  thank  the 
members  of  the  Program  Committee  for  their  hard  work  in  reviewing  and  selecting  the 
papers  that  appear  in  this  book.  We  are  especially  grateful  to  the  authors  who  submitted 
their  papers  to  this  conference  and  to  the  presenters  who  provided  the  substance  of  the 
meeting.  We  wish  to  thank  all  members  of  our  organizing  committee. 

Last  but  not  the  least,  we  hope  that  everybody  will  have  a good  time  in  Prague,  and  we  invite 
all  participants  for  the  next  edition  that  will  be  held  in  Lisbon,  Portugal. 


Miguel  Baptista  Nunes,  University  of  Sheffield,  United  Kingdom 
Maggie  McPherson,  University  of  Leeds,  United  Kingdom 
e-Learning  2013  Conference  Program  Co-Chairs 

Piet  Kommers,  University  of  Twente,  The  Netherlands 

Pedro  Isaias,  Universidade  Aberta  (Portuguese  Open  University),  Portugal 

Eva  Kasparova,  University  of  Economics,  Faculty  of  Business  Administration,  Prague,  Czech 

Republic 

MCCSIS  2013  General  Conference  Co-Chairs 

Prague,  Czech  Republic 
July  2013 


XIV 


PROGRAM  COMMITTEE 


E-LEARNING  CONFERENCE  PROGRAM  CO-CHAIRS 

Miguel  Baptista  Nunes,  University  of  Sheffield,  United  Kingdom 
Maggie  McPherson,  University  of  Leeds,  United  Kingdom 

MCCSIS  GENERAL  CONFERENCE  CO-CHAIRS 

Piet  Kommers,  University  of  Twente,  The  Netherlands 
Pedro  Isaias,  Universidade  Aberta  (Portuguese  Open  University),  Portugal 
Eva  Kasparova,  University  of  Economics,  Faculty  of  Business  Administration,  Prague, 

Czech  Republic 


E-LEARNING  CONFERENCE  COMMITTEE  MEMBERS 

Adamantios  Koumpis,  ALTEC  Software  S.  A.,  Greece 
Adel  Elsayed,  Retired,  University  Of  Bolton,  United  Kingdom 
Alexandra  Cristea,  University  Of  Warwick,  United  Kingdom 
Alexei  Tretiakov,  Massey  University,  New  Zealand 
Alla  Manako,  Irtc,  Ukraine 

Amali  Weerasinghe,  University  Of  Canterbury,  New  Zealand 
Amjad  Mahmood,  University  Of  Bahrain,  Bahrain 
Ana  Isabel  Vazquez  Martinez  , University  Of  Sevilla,  Spain 
Andreas  Bollin,  Klagenfurt  University,  Austria 
Andreas  Papasalouros,  University  Of  The  Aegean,  Greece 
Andrew  Lian,  Western  Illinois  University,  Usa 
Angelica  De  Antonio,  Universidad  Politecnica  De  Madrid,  Spain 
Ania  Lian,  Australian  Catholic  University,  Australia 
Annabell  Preussler,  University  of  Duisburg-Essen,  Germany 
Annette  Payne,  Brunei  University,  United  Kingdom 
Anouk  Gelan,  University  Hasselt,  Belgium 
Antoanela  Naaji,  Vasile  Goldis  West  University  Of  Arad,  Romania 
Antonio  Hervas- Jorge,  Universidad  Politecnica  De  Valencia,  Spain 
Antonio  Navarro,  Universidad  Complutense  De  Madrid,  Spain 
Antonio  Panaggio,  Italian  Ministry  Of  Education,  Italy 
Apostolos  Gkamas,  University  Ecclesiastical  Academy  of  Vella  of  loan,  Greece 
Avgoustos  Tsinakos,  Department  Of  Industrial  Informatics,  Greece 
Barbara  N.  Brecko,  European  Commission,  Spain 
Ben  Du  Boulay,  University  Of  Sussex,  United  Kingdom 


XV 


Benno  Volk,  ETH  Zurich  - Swiss  Federal  Institute  of  Technolog,  Switzerland 
Bettina  Pedemonte,  Institute  For  Educational  Technology  Itd-cnr,  Italy 
Branislav  Smitek  , University  Of  Maribor , Slovenia 
Carmen  Holotescu,  University  Politehnica  Timisoara,  Romania 
Cengiz  Hakan  Aydin,  Anadolu  University,  Turkey 
Cerstin  Mahlow,  University  of  Basel,  Switzerland 
Charalampos  Karagiannidis,  University  of  Thessaly,  Greece 
Charoula  Angeli,  University  Of  Cyprus,  Cyprus 
Chen  Xiaowu,  Beihang  University,  China 
Christine  Fevy,  Kaleidoscope  Feaming,  Usa 
Christos  Bouras,  University  Of  Patras,  Greece 
Claudia  De  Witt,  Femuniversitat  In  Hagen,  Germany 
Cleo  Sgouropoulou,  Technological  Educational  Institute  (tei)  Of  Athen,  Greece 
David  Guralnick,  Kaleidoscope  Feaming,  United  States 
Demetrios  Sampson,  University  Of  Piraeus,  Greece 
Diana  Perez  Marin,  Rey  Juan  Carlos  University,  Spain 
Dimitra  Pappa,  National  Center  For  Scientific  Research  - Demokrit,  Greece 
Dimitris  Fotiadis,  University  Of  loannina,  Greece 
Dirk  Ifenthaler,  Open  Universities  Australia,  Australia 
Egle  Butkeviciene,  Kaunas  University  Of  Technology,  Fithuania 
Elena  Mosa,  Indire,  Italy 
Eliza  Stefanova,  Sofia  University,  Bulgaria 
Emma  Briend,  University  Of  Fimerick,  Ireland 
Esma  Aimeur,  University  Of  Montreal,  Canada 
Esteban  Vazquez  Cano,  Spanish  National  University  Of  Distance  Education,  Spain 
Eugenia  Kovatcheva,  University  Of  Sofia,  Bulgaria 
Eva  Heinrich,  Massey  University,  New  Zealand 
Eva  Jereb,  University  Of  Maribor,  Slovenia 
Fawzi  Albalooshi,  National  Authority  of  Qualifications  and  Quality  A,  Bahrain 
Francesca  Pozzi,  Institute  Tecno logic  Didattiche  - CNR,  Italy 
Fridolin  Wild,  The  Open  University,  United  Kingdom 
Frina  Albertyn,  Eastern  Institute  Of  Technology,  New  Zealand 
Fuhua  Fin,  Athabasca  University,  Canada 
Gabriela  Grosseck,  West  University  Of  Timisoara,  Romania 
Gabriella  Dodero,  Free  University  Of  Bolzano  Bozen,  Italy 
Gary  Wills,  University  Of  Southampton,  United  Kingdom 
Geoffrey  Eautenbach,  University  Of  Johannesburg,  South  Africa 
George  Palaigeorgiou,  CSD,  AUTH,  Greece 
George  Papadourakis,  School  Of  Applied  Technology,  Greece 
George  Tsihrintzis,  University  Of  Piraeus,  Greece 
Gianni  Vercelli,  DIBRIS  - University  of  Genoa,  Italy 
Giovanni  Bonaiuti,  University  Of  Cagliari,  Italy 
Giuliana  Dettori,  ITD-CNR,  Italy 
Giuliano  Vivanet,  University  Of  Cagliari,  Italy 
Giusy  Cannella,  Indire,  Italy 

Gloria  Yi-ming  Kao,  National  Taiwan  University  Of  Science  And  Technolo,  Taiwan 
Guglielmo  Trentin,  Cnr  - Istituto  Tecnologie  Didattiche,  Italy 


xvi 


Hamid  Harroud,  Al-Akhawayn  University  In  Ifrane,  Morocco 
Helena  Bukvova,  WU,  University  of  Economics  and  Business  Vienna,  Austria 
Imed  Hammouda,  Tampere  University  Of  Technology,  Finland 
Ingo  Dahn,  University  Of  Koblenz-landau,  Germany 
loannis  Vardiambasis,  Technological  Educational  Institute  (TEI)  of  Crete,  Greece 
Ivan  Ganchev,  University  of  Eimerick,  Ireland 
Ivana  Simonova,  University  Of  Hradec  Kralove,  Czech  Republic 
Jaime  Ramirez,  Universidad  Politecnica  De  Madrid,  Spain 
Jan  Pawlowski,  University  of  Jyvaskyla,  Finland 
Jane  Sinclair,  University  Of  Warwick,  United  Kingdom 
Jarkko  Suhonen,  University  of  Eastern  Finland,  Finland 
Jekaterina  Buie,  Riga  Technical  University,  Fatvia 
Jerzy  Dabkowski,  Cracow  University  Of  Technology,  Poland 
Jesualdo  Breis,  University  Of  Murcia,  Spain 
Jesus  Boticario,  Universidad  Nacional  De  Educacion  A Distancia  (une,  Spain 
Jesiis  Sanchez  Allende,  Universidad  Alfonso  X El  Sabio,  Spain 
Jirarat  Sitthiworachart,  Walailak  University,  Thailand 
Johannes  Magenheim,  Universitaet  Paderbom,  Germany 
Jonatan  Castano  Munoz,  European  Commission,  Spain 
Jozef  Hvorecky,  City  University  Of  Seattle,  Slovakia 
Jozef  Simuth  , City  University  Of  Seattle,  Slovakia 
Juan  M.  Alducin-ochoa,  Universirty  Of  Sevilla,  Spain 
Juan  M.  Santos,  University  of  Vigo,  Spain 
Katerina  Kabassi,  TEI  of  Ionian  Islands,  Greece 
Kateryna  Synytsya,  Ukraine  International  Research  And  Training  Center,  Ukraine 
Katherine  Maillet,  Institut  Telecom  & Management  Sudparis,  France 
Kazunori  Nozawa,  Ritsumeikan  University,  Japan 
Kostas  Vassilakis,  Technological  Educational  Institution  Of  Crete,  Greece 
Earisa  Zaiceva,  Riga  Technical  University,  Eatvia 
Eeonardo  Garrido,  Tecnologico  de  Monterrey,  Mexico 
Eiodakis  George,  Technological  Educational  Institution  Of  Crete,  Greece 
Eorraine  Cleeton,  Walden  University,  Usa 
Euis  Alvarez-Gonzalez,  Universidad  Austral  De  Chile,  Chile 
Euis  Anido-Rifon,  University  Of  Vigo,  Spain 
Eynde  Tan,  National  Institute  Of  Education  , Singapore 
Maiga  Chang,  Athabasca  University,  Canada 
Mairtin  ODroma,  University  Of  Eimerick,  Ireland 
Malinka  Ivanova,  Technical  University  Of  Sofia,  Bulgaria 
Mandy  Rohs,  University  of  Duisburg -Essen,  Germany 
Manolis  Tsiknakis,  Forth,  Greece 
Manos  Varvarigos,  University  Of  Patras,  Greece 
Manuela  Repetto,  Institute  Per  Ee  Tecnologie  Didattiche  - Cnr,  Italy 
Marek  Stanuszek,  Cracow  University  Of  Technology,  Poland 
Maria  Moundridou,  School  of  Pedagogical  and  Technological  Education  , Greece 
Maria  Virvou,  University  Of  Piraeus,  Greece 
Marina  Ribaudo,  University  Of  Genova,  Italy 
Mario  Vacca,  "Ea  Sapienza"  University  of  Rome  , Italy 


xvii 


Martin  Ebner,  Graz  University  of  Technology,  Austria 
Martin  Wessner,  Fraunhofer  lESE,  Germany 
Martin  Wolpers,  Katholieke  Universiteit  Eeuven,  Belgium 
Martin  Elamas-Nistal,  University  Of  Vigo,  Spain 
Martine  Verjans,  Universiteit  Hasselt,  Belgium 
Michael  Paraskevas,  Computer  Technology  Institute  & Press,  Greece 
Michael  Piotrowski,  Eeibniz  Institute  of  European  History,  Germany 
Michail  Kalogiannakis,  University  of  Crete,  Greece 
Michalis  Xenos,  Hellenic  Open  University,  Greece 
Michel  Ebouyea,  University  of  Ea  Rochelle,  France 
Miguel  Zapata  Ros,  University  Of  Alcala  de  Henares,  Spain 
Mihaela  Dinsoreanu,  Technical  University  Of  Cluj-napoca,  Romania 
Mihai  Jalobeanu,  Universitatea  de  Vest , Romania 
Mike  Joy,  University  Of  Warwick,  United  Kingdom 
Mirjana  Ivanovic,  University  Of  Novi  Sad,  Serbia 
Mizue  Kayama,  Shinshu  University,  Japan 
Mohamed  Amine  Chatti,  RWTH  Aachen  University,  Germany 
Murat  Ataizi,  Anadolu  University,  Turkey 
Natalija  Proko^eva,  Riga  Technical  University,  Eatvia 
Nicola  Capuano,  Diem  - University  Of  Salerno,  Italy 
Niki  Eambropoulos,  University  Of  Patras,  Greece 
Nora  Eizenberg,  Universidad  TecnologicaNacional,  Argentina 
Oliver  Bohl,  University  of  Kassel,  Germany 
Panagiotes  Anastasiades,  University  Of  Crete,  Greece 
Panagiotis  Kampylis,  European  Commission  - Dg  Jrc  , Spain 
Paolo  Gentilini,  University  Degli  Studi  Di  Genova,  Italy 
Patrick  Blumschein,  University  of  Freiburg,  Germany 
Pavel  Rusakov,  Riga  Technical  University,  Eatvia 
Peter  Mikulecky,  University  of  Hradec  Kralove,  Czech  Republic 
Petra  Poulova,  University  Of  Hradec  Kralove,  Czech  Republic 
Pilar  Moreno  Diaz,  Alfonso  X El  Sabio  University,  Spain 
Rafael  Morales,  University  of  Guadalajara,  Mexico 
Rafael  Valencia,  Universidad  De  Murcia,  Spain 
Raul  Antonio  Aguilar  Vera,  Universidad  Autonoma  De  Yucatan,  Mexico 
Rolf  Granow,  Fachhochschule  Euebeck,  Germany 
Rosabel  Roig-Vila,  Universidad  de  Alicante,  Spain 
Rose  Eiang,  National  Institute  Of  Education  , Singapore 
Roza  Dumbraveanu,  The  State  Pedagogical  University  l.creanga,  Moldova,  Republic  Of 
Ruben  Edel  Navarro,  Universidad  Veracruzana,  Mexico 
Rune  Pettersson,  Institute  for  Infology,  Sweden,  Sweden 
Sandra  Schoen,  Salzburg  Research  Forschungsgesellschaft,  Austria 
Santiago  Mengual  Andres  , University  Of  Valencia,  Spain 
Sobah  A.  Petersen,  Sintef,  Norway 

Spiros  Sirmakesis,  Technological  Educational  Institutional  of  Messolo,  Greece 
Stamatina  Anastopoulou,  University  Of  the  Aegean,  Greece 
Susanne  Draheim,  University  Of  Hamburg,  Germany 
Telmo  Zarronandia,  Carlos  111  University,  Spain 


Thomas  Lehmann,  Albert-Ludwigs  University  of  Freiburg,  Germany 
Thrasivoulos  Tsiatsos,  University  Of  Patras,  Greece 
Tomas  Pitner,  Masaryk  University,  Czech  Republic 
Tomayess  Issa,  Curtin  University,  Australia 
Tomaz  Amon,  Center  For  Scientific  Visualization,  Slovenia 
Tuomo  Kakkonen,  University  Of  Joensuu,  Finland 
Ulrike  Hanke,  Universitaet  Freiburg,  Germany 
Vaiva  Zuzeviciute,  Magnus  University,  LITHUANIA 
Vassilis  Triantafillou,  Technological  Educational  Institution  Of  Mess,  Greece 
Veijo  Meisalo,  University  of  Helsinki,  Finland 
Veronika  Makarova,  University  Of  Saskatchewan,  Canada 
Werner  Beuschel,  IBAW  Brandenburg,  Germany 
Xiaokun  Zhang,  Athabasca  University,  Canada 
Yannis  Psaromiligkos,  Technological  Education  Institute  Of  Piraeus,  Greece 
Zinayida  Petrushyna,  Rwth  Aachen,  Germany 


KEYNOTE  LECTURES 


THOUGHTS  ON  THE  QUALITY  OF  LEARNING  IN  MOOCS 


Thomas  C.  Reeves,  Ph.D. 

Professor  Emeritus  of  Learning,  Design,  and  Technology 
College  of  Education 
The  University  of  Georgia 


ABSTRACT 

Massive  Open  Online  Courses  (MOOCs)  and  other  forms  of  open  education  are  viewed  as  positive 
disruptive  innovations  by  some  and  as  threats  to  traditional  education  by  others.  The  discord  over 
MOOCs  is  especially  vociferous  in  the  higher  education  sector.  This  paper  examines  the  evidence 
for  and  against  MOOCs,  especially  with  respect  to  the  quality  of  the  learning  accomplished  by 
participants.  The  paper  also  discusses  inconsistencies  in  assessment  in  MOOCs,  and  suggests 
appropriate  assessment  strategies  based  on  more  authentic  approaches.  At  the  conclusion  of  this 
paper,  recommendations  are  made  for  advancing  the  efficacy  and  impact  of  MOOCs  through 
educational  design  research. 
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ABSTRACT 

The  paper  deals  with  the  new  and  actual  approaches  in  E-learning.  It  seems,  that  the  period  of 
sophisticated  Learning  Management  System  (LMS)  is  over  and  the  potential  customers  = e- 
students  prefer  new  environment  for  their  learning  activities.  Interactivity  and  communication  tools 
are  considered  as  the  crucial  point  how  to  make  the  individual  e-study  more  open  and  users 
friendly.  Sharing  learning  experience,  external  study  materials  and  study  problems  facilitates  the 
study  processes  and  is  helping  to  manage  the  stress  in  the  learning  of  lonely  student. 

Keywords:  social  networks,  independent  study.  Learning  Management  Systems,  learning  and  teaching 
processes,  e-learning  environment. 
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Kettering  University,  USA  and  Institute  of  Mathematics  and  Computer  Science, 
Bulgarian  Academy  of  Sciences,  Bulgaria 


ABSTRACT 

The  beginnings  of  distance  education  date  back  over  one  hundred  years  ago  to  the  age  of 
correspondence  education  when  Frederick  Turner  ran  the  correspondence  history  program  at  the 
University  of  Wisconsin.  Resent  research  clearly  shows  that  students  who  learn  at  a distance  do  not 
learn  any  better  or  any  worse  than  the  traditional  students. 

Web  2.0  and  social  software  tools  have  tremendous  potential  to  remove  the  barriers  of 
distance  education.  The  social  dimension  of  Web  2.0  tools  has  already  begun  to  change  the 
traditional  paradigm  of  distance  education.  The  challenges  that  social  software  addresses  (meeting, 
building  community,  providing  mentoring  and  personal  learning  assistance,  working  collaboratively 
on  projects  or  problems,  reducing  communication  errors,  and  supporting  complex  group  functions) 
have  application  to  educational  use. 

In  the  tutorial,  we  identify  the  milestones  of  techniques  that  were  developed  to  meet  the 
needs  of  a new  type  of  learning  that  beneficially  exploits  the  domain  of  distance-learning  courses.  A 
methodology  for  converting  a regular  course  into  a distance  course  is  presented.  For  the  conversion 
process  we  use  the  ADDIE  technology.  The  ADDIE  (Analysis,  Design,  Development,  Implement, 
and  Evaluate)  model  is  the  generic  process  traditionally  used  by  instructional  designers  and  training 
developers.  The  five  phases  represent  a dynamic  and  flexible  guideline  for  building  effective 
training  and  performance  support  tools.  Topics  such  as:  develop  instructional  strategy,  develop  and 
select  instructional  materials;  design  and  conduct  formative  evaluation,  revise  instruction;  design 
and  conduct  summative  evaluations  are  also  presented.  Some  of  the  education  theories  such  as: 
theories  of  independence  and  autonomy,  theories  of  industrialization  of  teaching,  theories  of 
interaction  and  communication  are  discussed.  The  conversion  process,  the  technique  issues, 
requirements  and  conversion  steps  are  outlined.  Different  distance  learning  technologies  are  given. 

The  conversion  of  the  regular  course  “Web  technology”  into  distance  course  is  presented. 
“Web  technology”  is  a regular  course  that  was  converted  into  a distances  course,  which  we 
designed  and  implement  in  the  frame  of  the  “Master  of  Science  in  Information  Technology 
program”  in  the  Kettering  University.  The  course  is  based  on  video,  textbooks,  “Blackboard” 
system  based  information  and  e-mail  communications.  The  course  is  accompanied  by  a list  of 
books,  which  are  required  to  be  used  by  the  learner  to  prepare  for  each  session  as  well  as  a list  of 
recommended  books  and  papers,  which  discuss  various  issues  and  elaborate  on  the  topics. 

As  is  the  case  in  writing  a textbook,  the  development  of  a good  distance  education  course  is 
a long-term  process  of  trials  and  errors. 
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ABSTRACT 

Massive  Open  Online  Courses  (MOOCs)  and  other  forms  of  open  education  are  viewed  as  positive  disruptive  innovations 
by  some  and  as  threats  to  traditional  education  by  others.  The  discord  over  MOOCs  is  especially  vociferous  in  the  higher 
education  sector.  This  paper  examines  the  evidence  for  and  against  MOOCs,  especially  with  respect  to  the  quality  of  the 
learning  accomplished  by  participants.  The  paper  also  discusses  inconsistencies  in  assessment  in  MOOCs,  and  suggests 
appropriate  assessment  strategies  based  on  more  authentic  approaches.  At  the  conclusion  of  this  paper,  recommendations 
are  made  for  advancing  the  efficacy  and  impact  of  MOOCs  through  educational  design  research. 

KEYWORDS 

MOOC,  Quality,  Assessment,  Accreditation,  Educational  Design  Reseai'ch. 


1.  INTRODUCTION 

Quality  tends  to  fan  out  like  waves.  The  Quality  job  he  didn't  think  anyone  was  going  to  see  is  seen, 
and  the  person  who  sees  it  feels  a little  better  because  of  it,  and  is  likely  to  pass  that  feeling  on  to 
others,  and  in  that  way  the  Quality  tends  to  keep  on  going.  - Pirsig  (1974),  p.  357 

Since  late  2011,  MOOCs  (Massive  Open  Online  Courses)  have  been  the  subject  of  global  attention  among 
educators  and  the  public  at  large  as  well  as  the  target  of  large  financial  and  intellectual  property  investments 
by  some  of  the  most  prestigious  institutions  of  higher  education  around  the  world  (Daniel,  2012).  This  brief 
paper  is  intended  to  address  the  quality  of  MOOCs,  especially  with  respect  to  the  learning  accomplished  by 
those  who  “enroll”  in  them. 

The  concept  of  a MOOC  first  emerged  from  Canada  in  2008  when  George  Siemens  and  Stephen  Downes 
offered  a free  online  course  focused  on  “Cormectivism  and  Connective  Knowledge”  to  a few  fee-paying 
students  at  the  University  of  Manitoba  and  more  than  2,000  interested  learners  from  around  the  world  who 
“enrolled”  for  free.  The  MOOCs  offered  by  Siemens  and  Downes  and  others  have  been  referred  to  as 
cMOOCs  to  reflect  their  connectivist  and  constructivist  pedagogical  origins  whereas  the  MOOCs  offered  by 
people  and  entities  associated  with  certain  elite  U.S.  universities  (such  as  Stanford,  MIT,  and  Harvard)  have 
been  referred  to  as  xMOOCs  to  reflect  what  some  see  as  their  roots  in  behaviorist  or  “transmissionist” 
(teaching  by  telling)  pedagogy  and/or  their  stated  goals  of  making  profits  (Siemens,  2012). 

Sources  of  information  related  to  MOOCs  have  expanded  rapidly  over  the  past  two  years,  encompassing 
blogs  (such  as  http;//www.downes.ca/news/index.html  and  http;//www.elearnspace.org/blog/),  press  reports, 
and  a few  academic  journals.  The  Chronicle  of  Higher  Education,  the  weekly  newspaper  of  record  for  higher 
education  in  the  U.S.,  published  a special  Online  Learning  supplement  focused  on  MOOCs  in  October  2012. 
In  May  2013,  the  New  Yorker  magazine  published  a lengthy  examination  of  MOOCs  titled  “Laptop  U:  Has 
the  Future  of  College  Moved  Online?”  (Heller,  2013).  In  addition,  several  refereed  journals  have  issued  or 
will  soon  publish  special  issues  focused  on  MOOCs  (cf  Research  and  Practice  in  Assessment,  Journal  of 
Online  Learning  and  Teaching  (JOLT)  and  Distance  Education). 

Three  major  MOOC  providers  emerged  in  2012: 

• Coursera  (https://www.coursera.org/)  is  a for-profit  venture  founded  by  Andrew  Ng  and  Daphne 
Roller,  two  computer  science  professors  from  Stanford  University. 

• edX  (https://www.edx.org/)  was  established  by  the  Massachusetts  Institute  of  Technology  and 
Harvard  University  as  a non-profit  venture.  The  University  of  California-Berkeley,  the  University  of 
Texas  System,  and  other  highly  rated  universities  are  also  involved  in  this  MOOC  collaboration. 

• Udacity  (http://www.udacity.com/)  was  founded  as  a for-profit  corporation  by  Sebastian  Thrun, 
David  Stavens,  and  Mike  Sokolsky,  three  entrepreneurs  connected  to  Stanford  University. 
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In  addition  to  the  well  publicized  and  for  now  well-funded  enterprises  of  Udacity,  Coursera,  and  edX, 
there  have  been  many  other  MOOC  offerings.  For  example,  with  backing  from  BlackBoard,  an  international 
course  management  system  company,  Curt  Bonk  from  Indiana  University,  author  of  The  World  Is  Open: 
How  Web  Technology  Is  Revolutionizing  Education  (2009),  offered  a MOOC  titled  “Instructional  Ideas  and 
Technology  Tools  for  Online  Success”  (http://events.blackboard.com/open)  to  more  than  4,000  learners  in 
April/May  2012.  Professor  Paul  Kim,  Chief  Technology  Officer  and  Assistant  Dean  for  the  Stanford 
University  School  of  Education,  has  offered  a MOOC  titled  “Designing  a New  Learning  Environment” 
(http://venture-lab.org/education)  in  which  small  teams  of  participating  learners  collaborate  to  create  designs 
for  innovative  learning  environments. 

MOOCs  are  not  just  a North  American  innovation.  A project-based  learning  approach  was  employed  in  a 
MOOC  titled  “Learning  Design  for  a 21st  Century  Curriculum”  that  originated  from  the  Open  University  of 
the  United  Kingdom  (http://www.olds.ac.uk/)  from  January-March  2013.  In  March  2013,  Open  Universities 
Australia  released  its  Open2Learn  platform  (https://www.open2study.com/)  offering  MOOCs  in  topics 
ranging  from  anthropology  to  writing.  In  April  2013,  eleven  European  universities  launched  OpenUpEd 
(http://www.openuped.eu/),  a MOOC  platform  that  already  offers  more  than  40  courses  in  12  languages. 
Moessinger  (2013)  wrote  a three-part  article  surveying  MOOCs  around  the  world  that  includes  additional 
information  about  MOOCs  being  developed  in  other  parts  of  the  world,  including  Asia  and  South  America. 

Clearly  there  has  been  an  enormous  growth  of  the  MOOC  phenomenon.  MOOCs  are  “massive;”  tens  of 
thousands  of  people  have  signed  up  for  selected  MOOCs.  Most  MOOCs  are  “open”  in  the  sense  that  fees  are 
not  required,  although  costs  may  be  involved  for  specific  “premium”  services  (e.g.,  Coursera’s  “Signature 
Track  Verified  Certificate”  of  course  completion).  So-called  xMOOCs  are  not  open  in  the  same  sense  that 
cMOOCs  are  in  that  in  the  former  the  creators  of  the  content  retain  the  copyright  for  most  of  what  is  included 
in  the  learning  environment  whereas  in  the  connectivist  MOOCs,  content  is  freely  created  and  shared  by 
everyone  involved.  MOOCs  are  almost  completely  “online,”  although  in  some  cases,  learners  wishing  to  earn 
transferrable  credits  must  take  final  exams  and  other  assessments  at  proctored  physical  sites.  Whether  they 
are  “courses”  in  the  traditional  sense  of  this  hallowed  academic  construct  gets  to  the  crux  of  the  matter.  After 
all,  amidst  all  this  creative  effort  and  investment,  one  question  remains  inadequately  addressed:  “What  is  the 
evidence  that  high  quality  learning  is  taking  place  on  the  part  of  those  who  enroll  in  MOOCs?” 


2.  ASSESSMENT,  CERTIEICATION,  AND  ACCREDITATION 

Before  directly  addressing  the  evidence  of  high  quality  learning  question,  three  important  issues  must  first  be 
considered:  1)  assessment,  2)  certification,  and  3)  accreditation.  Assessment  is  the  process  of  determining 
what  people  are  learning  in  any  type  of  learning  environment  as  well  as  the  extent  to  which  they  are  learning. 
Traditional  assessments  including  tests  and  essays  whereas  alternative  assessments  include  portfolios  of 
cumulative  products  and  completion  of  authentic  tasks  (Oosterhof,  Conrad,  & Ely,  2008). 

Certification  refers  to  the  process  whereby  an  institution  of  higher  education  warrants  that  someone  has 
learned  and  issues  a credential  (e.g.,  a Bachelor  of  Arts)  as  evidence  of  that  learning  (Astin  & Antonio, 
2012).  For  most  people  formally  enrolled  as  students  in  courses  at  a traditional  university  or  college,  a 
transcript  of  courses  completed  and  the  grades  or  marks  achieved  in  those  courses  usually  represents 
certification.  When  enough  courses  have  been  completed  under  the  organizing  scheme  of  a program  of  study, 
a student  is  awarded  a degree  that  further  certifies  a given  level  of  achievement  (e.g.,  a Masters  of  Science  in 
Biology  degree).  Several  interesting  developments  related  to  certification  and  MOOCs  have  occurred  in  the 
last  twelve  months.  For  example,  Antioch  University,  a U.S.  institution  with  five  campuses  in  four  states  and 
a reputation  for  innovation,  announced  in  October  2012  that  it  has  reached  an  agreement  to  offer  official 
college  credits  for  completion  of  MOOCs  offered  by  Coursera  (DeSantis,  2012).  In  February  2013,  the 
influential  American  Council  on  Education  announced  that  it  had  determined  that  five  of  the  MOOCs  offered 
by  Coursera  were  worthy  of  transferrable  university  credit  (Kolowich,  2013).  Since  then,  many  individual 
universities  as  well  as  various  academic  consortia  have  announced  plans  to  accept  MOOC -based  credits. 
Legislators  in  two  U.S.  states,  California  and  Elorida,  have  been  especially  proactive  in  promoting  the  wider 
acceptance  of  MOOC  credits  by  their  state  university  systems  (Ebersole,  2013a). 
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Accreditation  is  the  process  whereby  an  external  agency  warrants  that  an  institution  is  worthy  of  offering 
and  awarding  degrees  at  various  levels  (Jung  Latchem,  2012).  Accreditation  is  an  important  process  in  that 
it  provides  evidence  that  a higher  education  institution  is  maintaining  sufficient  quality  with  respect  to  its 
various  missions  such  as  teaching,  research,  and  service.  In  light  of  criticism  that  accreditation  agencies  have 
focused  too  much  on  “inputs”  to  institutions  (physical  plant,  degrees  held  by  faculty,  books  in  libraries,  etc.), 
these  agencies  are  currently  being  pressed  to  base  their  judgments  on  outcomes  such  as  graduation  rates, 
career  success  of  graduates,  and  other  indicators  (LaCelle -Peterson  & Rigden,  2012). 

Although  certification  and  accreditation  are  important  factors  that  require  more  refinement  as  they  relate 
to  MOOCs,  assessment  is  the  issue  most  closely  linked  to  the  question  of  evidence  of  learning.  Ebersole 
(2013b)  highlighted  the  problem:  “At  a time  when  evidence  of  learning  is  increasingly  demanded  by 
accreditors  and  the  federal  government,  a determination  of  equivalency  in  instruction  alone  is  no  longer 
sufficient.  Valid,  secure  learning  outcome  assessment  must  now  be  part  of  the  equation  as  well.” 

Although  it  may  sound  cynical  to  some,  unless  something  has  been  assessed  within  the  context  of  a 
formal  learning  environment  (regardless  of  whether  it  is  online  or  not),  there  is  no  acceptable  foundation  for 
warranting  that  learning  has  occurred  (Reeves  & Okey,  1996).  A corollary  to  this  conjecture  is  that  for  the 
most  part,  people  engaged  in  formal  learning  fail  to  put  sufficient  effort  into  learning  something  unless  they 
know  that  it  is  going  to  be  assessed.  So  how  is  learning  currently  assessed  within  MOOCs?  More 
importantly,  how  can  assessment  of  learning  within  MOOCs  be  improved  to  the  point  that  a clear  answer  to 
the  question  of  learning  in  MOOCs  can  be  provided? 


3.  ASSESSMENT  PRACTICES  IN  MOOCS 


The  providers  of  and  participants  in  some  MOOCs,  especially  those  employing  the  cMOOC  model  based  on 
constructivist  learning  principles,  are  not  usually  concerned  with  assessing  participant  learning.  For  example, 
Kop,  Fournier,  and  Mak  (2011)  described  two  connectivist  MOOCs  originating  in  Canada  in  which  neither 
course  included  “formal  assessments  of  learning  outcomes  as  the  learning  objectives  for  each  learner  on  the 
MOOCs  was  different,  dependent  on  his  or  her  contexf’  (p.  74).  The  basic  philosophy  of  these  types  of 
MOOCs  is  a focus  on  enabling  opportunities  to  learn  as  well  as  to  contribute  to  the  learning  of  others. 

For  the  providers  and  sponsors  of  other  MOOCS,  including  many  of  the  xMOOC  variety,  who  are  more 
concerned  with  assessing  participant  learning,  the  options  for  assessing  student  learning  within  a MOOC  are 
not  unlike  the  methods  used  in  traditional  face-to-face  courses  as  well  as  online  and  blended  courses  (Rovai, 
2000),  although  there  are  arguably  more  complex  challenges  involving  closely  related  issues  such  as  cheating 
and  plagiarism  (Oliver,  2012;  Young,  2012).  The  type  of  assessment  methods  used  within  any  type  of 
learning  environment  should  be  aligned  with  other  important  components  of  the  learning  environment, 
especially  the  learning  objectives  and  the  learning  activities  that  purport  to  enable  students  to  accomplish  the 
objectives  (Reeves,  2006).  As  an  example  of  the  alignment  between  objectives  and  assessment  strategies. 
Table  1 lists  various  assessment  strategies  that  could  be  used  in  a MOOC  to  assess  different  levels  of 
cognitive  learning  represented  in  Bloom’s  updated  taxonomy  (Anderson  & Krathwohl,  2001). 


Table  1.  Assessment  Strategies  for  MOOCs  Aligned  with  Objectives. 


Cognitive  Process 

Objective 

Assessment  Strategy 

1 . Remembering 

1. 1 Recognizing 

1.2  Recaiiing 

List  the  terms  represented  by  the  acronym 
ADDIE. 

Oniine  quiz 

2.  Understanding 

2. 1 Interpreting 

2.2  Exemplifying 

2.3  Classifying 

2.4  Summarizing 

2.5  inferring 

2.6  Comparing 

2. 7 Explaining 

Ciassify  objectives  according  to  the  revised 
version  of  Bloom’s  Taxonomy. 

Oniine  quiz 

3.  Appiying 

3. 1 Executing 

3.2  Implementing 

Demonstrate  the  capacity  to  use  iinear 
equations,  specificaliy  finding  soiutions  through 
graphing,  eiimination,  or  substitution,  in  soiving 
word  probiems. 

Peer  assessment  of  probiem 
soiution  guided  by  rubric 
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Cognitive  Process 

Objective 

Assessment  Strategy 

4.  Analyzing 

4. 1 Differentiating 

4.2  Organizing 

4.3  Attributing 

Analyze  the  probable  causes  of  climate 
change. 

Essay  marked  by  instructors 
and  other  subject  matter 
experts 

5.  Evaluating 

5. 1 Checking 

5.2  Critiquing 

Write  a review  of  a film  for  submission  to  a 
scholarly  journal. 

Review  refereed  by  journal 
editors 

6.  Creating 

6. 1 Generating 

6.2  Pianning 

6.3  Producing 

Design  and  produce  an  open  education 
resource  (OER)  for  a specific  learning 
objective. 

OER  tested  with  and 
evaluated  by  actual  learners. 

Roberts  (2012)  prepared  one  of  the  only  available  reports  detailing  assessment  activities  within  a MOOC. 
In  May  and  June  2012,  the  Oxford  Centre  for  Staff  and  Learning  Development  (OCSLD)  at  Oxford  Brookes 
University  in  the  UK  offered  a MOOC  titled  “First  Steps  in  Learning  and  Teaching.”  Targeted  for  novice 
lecturers,  doctoral  students  with  teaching  roles,  and  people  moving  from  industry  or  other  sectors  into  higher 
education,  the  MOOC  initially  attracted  200  enrollees  from  24  different  countries.  Of  these,  60  participants 
remained  in  the  MOOC  for  the  entire  six  weeks,  and  14  of  these  participants  completed  the  assessment  tasks 
and  received  a certificate  in  recognition  of  their  achievement.  According  to  Roberts  (2012),  there  were  three 
components  in  the  assessment  scheme  applied  in  the  “First  Steps  in  Learning  and  Teaching”  MOOC: 

• Activity  1 (10  days,  individual/and  public  discussion)  Brief  public  professional  reflective  profile 
statement  shaped  by  Standard  Descriptor  One  of  the  UK  Professional  Standards  Framework  for 
Teaching  in  Higher  Education;  to  be  posted  by  Monday  28  May  for  peer  discussion  by  Wednesday 
30  May  [NOTE  extended  deadline  for  submission  Friday  1 June  for  feedback  by  Friday  8 June] 

• Activity  2 (1  week,  whole  cohort)  Collaborative  annotated  bibliography  of  works  that  have 
influenced  you  in  respect  of  the  scholarship  and  practice  of  teaching;  to  be  done  by  Wednesday  6 
June. 

• Activity  3 (2  weeks,  individuaFsmall  group)  Micro  teaching  session.  To  run  from  Wednesday  6 
June  to  Wednesday  20  June.  There  will  be  four  options  for  engaging  in  this  activity. 

Roberts  (2012)  reported  that  for  the  most  part,  this  assessment  scheme  worked  well.  However,  in  future 
iterations  of  this  MOOC,  he  suggested  that  the  first  two  activities  should  be  formative  in  nature  and  that  only 
the  third  task  (micro  teaching  session)  would  actually  be  marked.  While  it  is  easy  to  conceive  how  videos  of 
the  Micro  teaching  sessions  of  14  participants  could  be  marked  by  the  MOOC  leaders,  if  this  assessment 
strategy  was  taken  up  by  hundreds  of  learners,  major  difficulties  would  arise.  Scaling  up  project-based 
assessments  is  a major  challenge  facing  MOOC  developers. 

Another  challenge  rests  with  human  foibles.  Cheating  and  plagiarism  are  concerns  within  any  assessment 
scheme,  but  especially  for  MOOCs.  Oliver  (2012)  noted  that  Coursera  is  utilizing  the  crowd  sourcing 
strategy  of  peer  assessment  to  assess  the  learning  that  cannot  be  easily  marked  by  a computer  algorithm.  She 
questioned  this  strategy  and  stated  “crowd  sourcing  can’t  be  relied  upon  when  self-interest  is  at  play.”  Oliver 
promoted  “authentic  assessment”  strategies  whereby  the  methods  used  to  assess  learning  closely  approximate 
the  actual  activities  people  face  in  the  real  world.  Herrington,  Reeves,  and  Oliver  (2010)  described  examples 
of  how  online  higher  education  courses  have  been  designed  around  significant  authentic  tasks  in  which 
assessment  is  inherent  in  the  tasks.  This  is  essentially  the  strategy  that  was  used  in  the  aforementioned 
“Learning  Design  for  a 21st  Century  Currieulum”  MOOC  organized  by  the  Open  University  of  the  UK. 

Another  strategy  to  combat  cheating  in  MOOCs  is  to  employ  external  invigilation  or  proctored  exams. 
According  the  Clayton  (2012),  Pearson  Assessment  Centers  can  now  facilitate  proctored  tests  and  exams  in 
MOOCs  offered  by  Coursera,  edX,  and  Udacity.  Parry  (2012)  reported  that  an  exam  proctored  by  Pearson  for 
Udacity  MOOCs  will  cost  the  learner  89  U.S.  dollars.  Students  passing  exams  at  Pearson  Assessment  Centers 
will  receive  certificates  indicating  that  their  assessments  were  proctored.  Learners  armed  with  such 
certificates  may  be  able  to  convince  the  universities  in  which  they  are  enrolled  to  accept  MOOC  participation 
for  credit  towards  a degree.  Schemes  for  proctoring  assessments  within  MOOCs  are  likely  to  proliferate  if 
MOOCs  continue  to  expand  at  their  current  rate. 
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4.  IMPROVING  LEARNING  ASSESSMENT  IN  MOOCS 

The  current  options  for  assessing  learning  in  MOOCs  are  similar  to  those  that  can  be  found  in  other  delivery 
systems  for  higher  education  such  as  face-to-face  courses,  blended  courses,  or  totally  online  courses. 
Regardless  of  the  type  of  delivery  system,  assessment  strategies  must  be  carefully  aligned  with  other 
components  of  the  learning  environment,  specifically,  the  objectives,  content,  learning  design/learning 
activities,  the  roles  of  the  instructor,  the  roles  of  the  learners,  and  the  technological  affordances  inherent  in 
the  delivery  system  (Reeves,  2006).  Misalignment  of  these  factors  is  a frequent  criticism  leveled  at  higher 
education  because,  although  most  programs  and  courses  will  state  lofty  goals  and  objectives,  they  rarely 
match  these  with  relevant,  reliable,  and  valid  assessment  strategies. 

Universities  offering  MOOCs  or  accepting  MOOC -based  credits  must  confront  issues  related  to  cheating, 
plagiarism,  and  other  human  failures  that  threaten  to  undermine  the  assessment  strategies  and  related  issues 
such  as  certification  and  accreditation.  The  incorporation  of  authentic  tasks  (Herrington  et  al.,  2010)  into 
MOOCs  addresses  alignment  issues  as  well  as  cheating  and  plagiarism.  One  of  the  inherent  advantages  of 
authentic  tasks-based  assessment  is  that  real  world  tasks,  especially  when  the  products  or  services  that 
learners  produce  or  provide  have  real  audiences  such  as  consumers  or  clients,  are  difficult  to  plagiarize. 

Authentic  tasks-based  assessment  should  be  the  hallmark  of  MOOCs  that  purport  to  certify  that 
participants  have  really  learned  higher-order  knowledge,  skills,  attitudes,  and  intentions.  For  example, 
imagine  a MOOC  for  higher  education  instructors  focused  on  alternative  assessment  methods.  The  goals  of 
this  MOOC  would  be  to  enable  instructors  to  incorporate  the  most  effective  types  of  assessment  methods 
within  their  courses.  Such  a MOOC  could  be  designed  to  present  a series  of  engaging  video  lectures  about 
different  assessment  practices.  Videos  of  these  various  methods  actually  being  applied  in  different  types  of 
courses  could  be  included  as  well  as  videos  of  instructors  and  students  discussing  the  pros  and  cons  of 
various  assessment  strategies.  Opportunities  to  experience  examples  of  different  assessment  strategies  could 
also  be  incorporated  into  the  MOOC,  such  as,  stealth  assessments  embedded  in  serious  games  (Shute,  2011) 
or  rigorous  multiple-choice  exams  that  have  been  carefully  validated  (Ebersole,  2013a). 

However,  the  primary  learning  task  in  the  proposed  MOOC  would  be  to  work  in  a small  team  to  develop 
an  alternative  assessment  scheme  for  an  instructor  who  desires  to  enhance  the  assessment  of  learning  in  one 
of  his/her  specific  courses  and  therefore  has  signed  up  in  the  MOOC  to  serve  as  a real  world  client.  Rubrics 
would  be  provided  to  allow  the  clients  as  well  as  the  MOOC  learners  to  judge  the  quality  of  the  alternative 
assessments  developed,  and  the  MOOC  providers  would  apply  similar  rubrics  to  the  assessment  of  a random 
sample  of  the  assessment  schemes  developed  by  the  teams.  Oh  and  Reeves  (in  press)  describe  how  a similar 
authentic  tasks  design  was  applied  to  an  online  course  focused  on  E-Learning  Evaluation,  and  Herrington  and 
Herrington  (1998)  described  how  authentic  assessments  have  been  applied  in  traditional  and  online  courses. 

Why  aren’t  more  MOOCs  and  other  higher  education  courses  designed  around  authentic  tasks?  It  may 
well  be  that  traditional  instructional  design  approaches  are  not  adequate  for  the  development  of  the  types  of 
authentic  tasks-based  MOOCs  recommended  here  (Oh  & Reeves,  2010).  Instead  educational  design  research 
(McKenney  & Reeves,  2012)  should  be  applied  to  the  development  of  MOOCs  if  they  are  to  be  designed  and 
implemented  in  such  a way  that  clear  evidence  of  learning  is  demonstrated.  Developing  a MOOC  that 
provides  unambiguous  evidence  of  participant  learning  is  the  kind  of  “wicked  problem”  that  does  not  yield 
easily  to  the  application  of  simple  design  heuristics  used  by  most  instructional  designers  (Krause,  2012). 
Wicked  problems  are  the  kinds  of  complex  problems  confronted  in  design  fields  such  as  architecture, 
engineering,  public  health,  and  education  where  human  actions  and  interactions  are  often  unpredictable 
(Margolis  & Buchanan,  1995). 

When  it  comes  to  innovative  solutions  to  complex  educational  problems,  researchers  have  primarily 
focused  on  the  question  “What  works?”  with  respect  to  teaching  and  learning,  with  generally  poor  outcomes 
(Schoenfeld,  2006).  Educational  design  research  fundamentally  changes  the  focus  of  research  from  “what 
works?”  questions  to  “how  can  we  make  this  work  and  why?”  intentions.  It  can  be  safely  predicted  that  if 
experiments  are  done  at  this  time  to  compare  MOOCs  with  traditional  higher  education  courses  the  most 
common  finding  will  be  “no  significant  differences”  (Reeves,  2011).  The  irony  of  these  ill-fated  comparisons 
is  that  neither  the  MOOCs  nor  the  traditional  courses  have  been  optimized  to  support  student  learning. 

The  twofold  goals  of  educational  design  research  are  first  to  solve  real  world  problems  through  the 
development  of  effective  interventions  (e.g.,  a MOOC  in  which  the  evidence  that  participants  have  learned  is 
unassailable)  and  second  to  reveal  reusable  design  principles  (e.g.,  how  to  design  authentic  tasks  in  which  the 
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assessment  of  learning  is  inherent).  MOOCs  developed  through  educational  design  research  protocols  would 
be  designed,  tested,  adopted,  implemented,  re-tested  and  refined  in  authentic  settings  iteratively  over  the 
lifespan  of  the  learning  environment.  Large-scale  MOOC  providers  such  as  Coursera,  Udacity,  and  edX 
could  be  appropriate  laboratories  for  conducting  the  kinds  of  educational  design  research  recommended  here. 
The  establishment  of  the  MOOC  Research  Initiative  (http://www.moocresearch.com/)  under  the  leadership  of 
MOOC  pioneer  George  Siemens  at  Athabasca  University  is  a hopeful  sign  that  a more  effective  research 
agenda  may  be  emerging  for  the  MOOC  phenomenon. 

As  a still  emerging  research  genre  (McKenney  & Reeves,  2013),  educational  design  research  applied  to 
MOOCs  and  other  types  of  open  educational  resources  will  demand  long-term  in-depth  collaboration  among 
researchers,  instructors,  and  learners.  It  will  require  a strong  commitment  to  creative  design  and  rigorous 
evaluation  work  that  will  last  many  months,  even  years.  However,  this  does  not  mean  that  any  given  MOOC 
must  wait  to  be  perfected  before  it  is  launched.  Educational  design  research  is  driven  by  the  best  upfront 
analysis  and  design  possible  along  with  creative  design,  but  further  testing  and  revision  of  prototype 
interventions  take  place  early  and  often. 

McKenney  and  Reeves  (2012)  described  educational  design  research  as  occurring  across  three  iterative 
phases:  1)  analysis  and  exploration,  2)  design  and  construction,  and  3)  evaluation  and  reflection.  Based  on 
my  personal  experience  as  a guest  instructor  in  three  MOOCs  and  as  a learner  in  three  different  MOOCs,  it 
appears  that  most  MOOCs  are  not  being  developed  using  educational  design  research  methods,  but  rather  by 
using  some  sort  of  rapid  prototyping  approach  that  emphasizes  media  production  (especially  for  videos)  over 
issues  such  as  alignment  across  pedagogical  dimensions  and  rigorous  assessment.  The  mindset  seems  to  be 
“build  it  and  they  will  come,”  whereas  what  is  really  needed  is  “design  it,  build  it,  implement  it,  evaluate  it, 
reflect  on  it,  refine  it,  implement  it  again,  evaluate  it  again,  reflect  on  it  again,  refine  it  again,  and  so 
on”... continuing  this  process  until  optimal  outcomes  are  realized  or  the  audience  for  the  MOOC  is  exhausted. 


5.  THE  REALITY  OE  LEARNING  AND  TEACHING  IN  MOOCS 

My  own  learning  experiences  in  MOOCs  have  been  disappointing  thus  far,  partly  the  result  of  poorly 
designed  learning  interactions  in  two  of  the  MOOCs  and  partly  my  own  fault  in  the  one  well-designed 
MOOC  in  which  I simply  could  not  keep  up  with  the  workload  during  the  last  two  weeks  of  the  six-week 
course  because  of  a heavy  travel  schedule.  My  “teaching”  experiences  in  three  MOOCs  have  also  left  me 
underwhelmed,  primarily  because  I have  failed  to  experience  any  real  connection  with  the  learners  in  these 
open  environments.  This  again  may  be  my  fault  because  while  some  of  the  other  learning  facilitators  in  these 
MOOCs  appear  to  utilize  every  social  media  tool  on  the  planet  to  “connect”  with  the  learners,  I am  still 
functioning  with  only  email,  discussion  forums,  Google  docs,  and  the  occasional  teleconference  via  Skype  or 
Google  Hangout. 

Setting  aside  my  personal  experience,  I believe  that  it  would  be  difficult  for  anyone  to  argue  that 
substantive  learning  is  occurring  within  most  MOOCs  or  for  most  learners.  The  extremely  high  attrition  rates 
in  MOOCs  attest  to  that  (Daniel,  2012).  But  when  learning  occurs,  it  is  because  the  MOOC  is  well  designed 
and  because  the  learner  is  willing  to  invest  the  necessary  effort  and  time  to  engage  in  the  learning  activities 
fully.  Both  of  these  must  occur.  We  cannot  fool  ourselves  and  imagine  that  we  can  design  MOOCs  wherein 
people  can  learn  effortlessly.  Learning  is  difficult,  developmental,  and  requires  time  and  commitment. 

Carey  (2013),  an  American  expert  on  higher  education  policy  from  the  New  America  Foundation, 
recently  wrote  one  of  the  most  useful  and  detailed  descriptions  of  what  it  is  really  like  to  learn  in  a MOOC. 
Carey  enrolled  in  MIT  7.00x,  an  introductory  biology  course,  one  of  the  first  courses  offered  by  the  Harvard- 
MIT  venture  edX.  Professor  Eric  Lander  who  was  a leader  of  the  Human  Genome  Project  taught  the  course. 
Carey  describes  how  he  invested  fifteen  hours  per  week  for  fifteen  straight  weeks  in  intensive  studying, 
managing  to  earn  a respectable  87%  score  on  the  rigorous  exams. 

This  level  of  effort  is  what  undergraduate  students  should  be  investing  in  every  one  of  their  undergraduate 
courses.  In  the  USA,  a typical  three-credit  semester  length  course  should  involve  three  hours  per  week  in 
class  or  labs,  and  another  9-12  hours  per  week  in  studying  and  relevant  learning  activities.  But  the  reality  is 
that  university  students  are  investing  much  less  time  in  their  academic  studies  than  this.  According  to 
Babcock  and  Marks  (2011),  “Full-time  students  allocated  40  hours  per  week  toward  class  and  studying  in 
1961,  whereas  by  2003  they  were  investing  about  27  hours  per  week”  (p.  468).  Ten  years  on,  it  is  likely  that 
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the  number  of  hours  spent  studying  has  declined  even  more,  and  as  Carey  (2013)  concluded,  “For  many 
students,  full-time  college  has  become  a part-time  job.”  What  is  especially  ironic  about  this  is  that  while  the 
time  invested  in  learning  has  steadily  declined,  the  grades  awarded  to  students  have  steadily  increased 
(Rojstaczer  & Healy,  2012).  Clearly,  the  professoriate,  at  least  in  the  USA,  is  expecting  less  effort  and  giving 
more  rewards  to  students  than  ever  before.  This  is  not  surprising  given  that,  first,  the  reward  systems  for  the 
professoriate  generally  value  research  and  external  grants  above  teaching,  and,  second  curi'icular  alignment  is 
rarely  scrutinized  in  any  serious  way.  Developing  rigorous,  academically  sound  MOOCs  through  rigorous 
educational  design  research  may  provide  a mechanism  for  turning  these  imbalances  around. 


6.  CONCLUSION 

Although  the  jury  is  still  out  whether  MOOCs  and  other  forms  of  open  education  will  fundamentally 
transform  higher  education  as  we  know  it,  the  need  to  transform  academe  cannot  be  doubted.  As  the  costs  of 
higher  education  have  far  outpaced  inflation  in  other  sectors  (Altbach,  Reisberg  & Rumbley,  2010)  and 
student  debt  grows  to  unacceptable  amounts,  at  least  in  the  USA,  something  must  fundamentally  change. 
There  is  much  weeping  and  gnashing  of  teeth  among  academics  over  what  they  see  as  the  commercialization 
of  higher  education  inherent  in  online  learning.  Rather  than  continuing  to  wail  over  our  fare,  we  in  the 
professorate  need  to  seize  the  moment  and  engage  in  socially  responsible  research  and  development  to 
improve  the  quality  and  impact  of  higher  education.  Instead  of  conducting  any  more  studies  to  compare 
classroom  learning  with  online  learning,  we  need  to  focus  on  making  online,  blended,  and  traditional 
classroom  education  more  effective  through  educational  design  research.  We  must  strive  to  assess  what  is 
important  to  learn  rather  than  what  is  easy  to  measure.  Pirsig  (1974)  wrote,  “Quality  tends  to  fan  out  like 
waves.”  This  is  true,  but  the  wave  must  start  somewhere.  It  begins  with  us. 
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ABSTRACT 

The  paper  deals  with  the  new  and  actual  approaches  in  E-leaming.  It  seems,  that  the  period  of  sophisticated  Learning 
Management  System  (LMS)  is  over  and  the  potential  customers  = e-  students  prefer  new  environment  for  their  learning 
activities.  Interactivity  and  communication  tools  are  considered  as  the  crucial  point  how  to  make  the  individual  e-study 
more  open  and  users  friendly.  Sharing  learning  experience,  external  study  materials  and  study  problems  facilitates  the 
study  processes  and  is  helping  to  manage  the  stress  in  the  learning  of  lonely  student. 
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1.  INTRODUCTION 

Actual  ways  of  E-learning  applications  are  very  different  and  many  users  of  Learning  Management  Systems 
(LMS)  and  many  course  authors  are  working  on  the  base  of  individual  experience.  Variety  of  LMS 
applications  and  lack  of  pedagogical  qualification  for  management  of  training  and  course  packages  writing 
lead  to  the  problematic  results.  Basic  theoretical  background  for  each  author  or  provider  of  E-learning 
(having  ambition  to  manage  on-line  events  in  E-learning)  could  be  taken  from  the  principles  of  distance 
education  (DE).  The  core  principle  of  DE  is  in  the  separation  of  the  teaching  and  learning  processes. 

As  the  new  trends  in  E-leaning  environment  are  considered  programmes  recognised,  like  social  networks. 
There  are  not  accepted  social  nets  like  Eacebook,  Linkedin  or  Yahoo  tools  (etc.  of  course).  There  is  a lot  of 
different  tools  at  the  market  for  networking  and  some  of  them  were  already  tested  for  the  purposes  of 
E-learning.  Some  of  universities  in  EU  started  already  to  test  this  new  opportunities  with  the  relative  success. 
To  sense  of  this  contribution  is  in  complex  approach  and  discussion  about  the  advantages  and  disadvantages 
of  this  new  learning  environment. 

Before  we  start  to  comment  this  new  environment  is  necessary  to  be  sure  and  understand  well  the 
complexity  and  evolution  from  the  simple  E-course  to  the  interactive  game  and  it  could  be  useful  to  describe 
categorisation  of  E-courses.  Contemporary  there  is  less  attention  done  on  the  pedagogical  components  and 
measurable  achievement  and  more  attention  is  devoted  to  IT  features,  content  and  testing.  Just  wrong  way  of 
course  objectives  definition  and  blind  using  ways  of  testing  (offered  by  LMS,  mostly  without  any  link  to  the 
measured  real  change  of  knowledge,  skills  and  competences)  brings  a lot  of  problems  and  in  fact  decrease 
substantially  the  effectiveness  of  E-courses  generally. 
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2.  CLASSIFICATION  OF  E-LEARNING  COURSES 

2.1  Simple  E-transfer  of  Information 

This  type  of  distribution  of  information  is  well  known  and  used  by  many  users.  For  application  of  such  a type 
of  sharing  information  and  their  distribution,  there  are  often  used  conventional  E-mails  with  the  list  of  target 
group.  There  is  not  confirmation,  that  the  receiver  have  read  the  text  (confirmation  of  opening  is  not 
information  about  the  reading!)  and  also  there  is  no  information  about  the  understanding  of  the  message. 

2.2  E-transfer  of  Information  with  Confirmation  of  Acceptance 

This  is  higher  level  of  E-communication,  because  there  is  usually  any  legal  measure  how  to  ensure  and 
confirm  the  acceptance.  The  signature  or  formal  declaration  of  acceptance  is  requested.  This  type  of 
E-transfer  is  used  for  purposes  of  responsibility  or  if  there  is  expected  transfer  of  this  knowledge  to  the  third 
person. 

2.3  E-transfer  of  Information  with  Confirmation  of  Understanding 

In  certain  cases  the  distributor  of  the  information  needs  to  be  sure,  that  content  of  the  message  was  fully 
understood  and  that  the  target  receiver  is  able  apply  new  information.  Solution  of  this  request  could  be  for 
example  in  the  simple  and  closed  test.  The  risk,  that  test  will  be  fulfilled  by  other  person  is  low. 

2.4  Knowledge  Based  (Informative)  E-Course  - Short 

This  is  typical  E-course  with  testing  and  assessment.  This  type  of  E-course  is  easy  to  prepare  for  potential 
authors  and  for  potential  receivers  is  easy  to  pass.  There  are  a lot  of  good  examples.  The  crucial  condition  is 
that  author  is  able  to  define  properly  learning  objective  and  learning  outcomes.  Also  the  appropriate  way  of 
assessment  according  to  the  right  identification  of  the  achieved  learning  change  is  necessary. 

2.5  Knowledge  Based  (Informative)  E-Course  - Long 

The  students  of  long  courses  have  the  problem  with  the  sustainable  motivation.  The  solution  is  in  the 
scheduling  and  separation  of  assessments  in  time.  Usually  is  applied  continuing  and  final  assessment  as  the 
tool  for  keeping  the  student  active  during  the  whole  E-course.  The  value  of  the  marking  from  the  continuing 
and  final  assessment  depends  on  the  type  of  course.  Definition  of  learning  outcomes,  objective  and  expected 
change  is  requested  as  well.  Both  this  type  knowledge  oriented  courses  could  well  follow  Bologna  process. 

2.6  Competence  Based  (Eormative)  E-Course  - Short 

Eormative  training  is  more  complicated  and  also  more  complex  education.  The  final  objective  is  to  change 
attitude,  development  of  the  new  skills  or  competences.  In  principle  it  needs  more  emotional  involvement  of 
student  and  more  interactivity  in  the  teaching/learning  processes.  The  claims  on  the  course  scenario  and 
selective  ways  of  assessment  are  high.  Still  the  development  of  such  a course  needs  deep  pedagogical 
knowledge  and  competences  of  its  creator.  Including  of  formative  training  into  the  E-courses  needs 
professional  approaches  and  psychological  and  educational  advisory.  Good  examples  from  DE  show,  that 
there  is  necessary  to  apply  less  leadership  and  more  independence  in  the  learning  process. 

2.7  Competence  Based  (Eormative)  E-Course  - Long 

Competence  based  courses  are  usually  made  in  the  shape  of  intensive  face  to  face  training  in  smaller  groups. 
Mostly  is  this  type  of  courses  targeted  on  the  middle  and  top  managers.  The  request  of  the  big  companies  is 
to  decrease  the  time  consumption  of  participants  and  to  facilitate  the  process  of  learning.  Top  managers  are 
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special  category  of  trainees  because  of  their  overloaded  time  schedule.  This  type  of  courses  is  usually 
prepared  as  tailored  course  and  it  means  that  it  is  expensive  and  it  needs  enough  time  for  preparatory.  As  a 
part  of  competence  based  courses  could  be  involved  also  managerial  on-line  games.  As  the  special  case,  there 
is  possible  to  engage  role  play  game. 


3.  APPLICATION  OF  SOCIAL  NETWORKING  IT  TOOLS  IN 
E-LEARNING 

Looking  at  the  above  mentioned  classification  we  can  clear  see,  that  for  more  simple  e-courses  there  is  not 
necessary  to  have  a special  management  tool  and  we  can  use  any  simple  IT  communication  system  (like  i.e. 
Outlook)  to  realise  it.  On  the  other  side  namely  the  formative  courses  need  more  interactivity  and  student 
collaboration  generally. 

Actual  experience  with  social  network  programmes  in  E-learning  has  faced  some  problems  of  this 
solution: 

1 . Students  like  it  on  the  very  beginning  of  the  e-course,  but  later  their  enthusiasm  is  dropping  down 
and  together  with  the  study  problems,  their  interest  to  communicate  with  other  students  is  not 
significantly  higher. 

2.  Insertion  of  files  into  the  social  networking  IT  programmes  is  not  so  easy,  like  in  the  conventional 
LMS. 

3.  The  management  of  e-course  at  the  social  network  is  highly  demanding  for  the  teacher  or 
administrator 

4.  usually  there  is  not  administrative  and  assessment  system  included. 

5.  compatibility  with  the  information  systems  of  universities  is  low  or  missing 

Speaking  about  these  problems  we  could  be  confused  if  this  is  the  right  way  of  the  new  approaches  to 
e-learning  and  if  it  could  be  useful  to  insert  the  implementation  of  such  a system  into  the  university  strategy 
plans.  The  answer  is  not  easy  to  be  done,  because  there  are  a lot  of  conditions,  which  must  be  accepted  before 
we  start  to  teach  at  the  social  networks. 

The  application  of  social  networks  IT  programmes  for  education  is  in  the  process  of  testing  and  looking 
for  the  right  composition  between  the  free  and  confidential  communication,  technical  solutions  related  to  the 
insertion  of  the  different  types  of  the  study  materials  and  compatibility  of  the  system  with  existing 
information  systems  at  the  universities.  We  are  now  in  the  situation,  that  there  is  first  necessary  to  answer 
some  questions  before  we  start  to  realise  this  type  of  innovative  educational  approach. 


4.  CONCLUSIONS 

Despite  of  the  fact,  that  former  paragraph  is  slightly  negative  there  is  the  big  future  in  changing  of  the  study 
environment  generally.  First  of  all  the  generation  of  the  new  learners  is  much  more  open  and  familiar  in 
using  virtual  environment  to  fulfil  their  needs.  Second  the  technical  problems  will  be  solved  soon,  in  fact 
immediately  after  definition  of  the  users  needs.  Social  networking  for  education  is  very  clear  and  visible 
challenge  for  all  educational  institutions  and  teacher,  how  to  change  steady  approaches  and  frozen  habits  and 
meanings  about  the  study  way. 

Lets  have  a positive  look  at  this  new  emerging  opportunities  and  lets  open  our  minds  to  the  new 
challenging  solutions 
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ABSTRACT 

This  paper  presents  trial  evaluation  of  a visual  computer  simulator  in  2009-2011,  which  has  been  developed  to  play  some 
roles  of  both  instruction  facility  and  learning  tool  simultaneously.  And  it  illustrates  an  example  of  Computer  Architecture 
education  for  University  students  and  usage  of  e-Learning  tool  for  Assembly  Programming  in  order  to  realize  effective 
and  efficient  ICT-based  Practical  Information  Education.  Evaluation  for  the  visual  simulator  has  been  carried  out  not  only 
as  an  instruction  facility  but  also  as  a self-learning  tool.  The  simulator  can  be  used  by  teachers  in  their  classroom  lectures 
as  well  as  by  students  for  their  making  reports  (essays).  Such  evaluated  results  are  based  on  viewpoints  from  instruction 
of  internal  structure  and  behaviour  in  vonNeumann  computer,  Assembly  Programming  exercise,  some  essays  by  students 
with  the  simulator,  and  relationship  between  student's  examination  scores  and  usage  of  the  simulator.  Second  half  of  the 
paper  describes  how  to  organize  multiple  server  system  for  collaborative  learning  with  the  visual  simulator. 


KEYWORDS 

Collaborative  learning,  multiple  server,  system,  qualitative  and  quantitative  evaluation. 


1.  INTRODUCTION 

People  of  the  world  say,  an  ICT-based  Education  has  become  widely  accepted  from  elementary  schools  to 
higher  educational  institutes.  Even  in  Japan,  Informatics  Eundamentals  has  been  included  as  one  of  the 
menus  of  University  Entrance  Examinations.  It  becomes  more  and  more  important  for  pupils  and  students  to 
learn  computer-related  subjects  even  in  not  higher  education. 

Many  instructors  and  researchers  have  designed  and  developed  some  kinds  of  computer  simulators 
because  they  seem  to  be  one  of  the  effectively  educational  tools  for  the  lecture  of  Computer  Architecture  and 
Information  Processing.  Some  of  the  famous  simulators  are  introduced  in  the  fifth  section  (parts  of  Related 
Works).  Such  simulators  can  be  used  in  the  practical  education.  Several  of  their  simulators  have  been  able  to 
illustrate  how  a computer  works  graphically  and  additionally  provide  some  kinds  of  programming  exercise 
environment.  Some  of  them  have  been  used  as  visual  educational  tools  of  instructors  for  their  learners  in 
classroom  lectures.  And  others  have  been  utilized  as  e-Learning  tools  for  Programming  Exercise  through 
even  after  school. 

It  is  clearly  recognized  to  be  useful  and  efficient  that  such  simulators  have  been  used  in  the  actual  and 
practical  education  for  more  than  decade  years  and  they  have  played  ones  of  significant  roles  in  Computer 
Architecture  and  its  related  lectures.  Although  such  simulators  have  been  providing  good  effects  for 
Computer  Architecture  education,  there  are  only  a few  reports  to  evaluate  the  simulators  in  the  practical 
education  and  estimate  their  effects  in  qualitative  and  quantitative  viewpoints.  So  we  have  described  trial 
evaluation  of  the  simulator  in  qualitative  approach  as  well  as  quantitative  one.  The  former  is  based  on 
students’  feedback  and  the  later  is  performed  by  statistical  analysis. 

Collaboration  and/or  collaborative  learning  are  important  for  users  to  work  together  in  some  kind  of 
effective/efficient  environment.  We  have  tried  to  organize  multiple  server  system  for  collaborative  learning 
and  have  implemented  collaborated  learning  environment  with  visual  computer  simulator.  We  will  introduce 
how  to  organize  multiple  server  system  for  collaborative  learning. 
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This  paper  presents  an  example  case  of  Computer  Architecture  including  usage  of  our  visual  computer 
simulator  in  the  next  (second)  section.  It  demonstrates  the  2009-201 1 evaluation  of  our  simulator  in  the  third 
section.  The  fourth  section  of  the  paper  introduces  organization  and  usage  of  multiple  server  system  for 
collaborative  learning.  The  fifth  section  mentions  the  related  works  about  our  research.  And  finally  the  paper 
summarizes  its  conclusion  in  the  last  (sixth)  section. 


2.  A VISUAL  COMPUTER  SIMULATOR 

A simulator  called  VisuSim  is  implemented  as  a pure  Java  program  to  provide  two  kinds  of  entries  for  both 
Java  applet  code  and  Java  application  one.  Each  can  be  selected  automatically  to  invoke  the  suitable  mode  of 
Java  program.  Namely,  for  example,  VisuSim  recognizes  its  invoking  environment  and  decide  to  execute  as  a 
Java  applet  in  the  environment  of  browser  or  to  work  as  a Java  application  in  the  environment  where  the  Java 
VM  executes  in  the  DOS  prompt  of  Windows,  in  the  command  interpreter  of  Linux  and  so  on.  A necessary 
condition  is  only  to  equip  the  Java  VM  prepared  to  execute  the  simulator.  So  it  is  very  useful  because  any 
executable  environment  will  do,  just  like  Windows,  Linux  and/or  Macintosh. 

There  are  both  sides  of  views  for  the  visual  simulator  as  are  summarized  below.  The  first  one  is  a tool  to 
visualize  an  internal  structure  and  behaviour  of  computer.  It  is  useful  for  teachers  to  show  students  how  a 
computer  works  graphically.  The  second  is  an  e-Learning  tool  to  provide  an  environment  for  Assembly 
Programming  exercise. 

2.1  Visualization  Tool 


VisuSim  has  8 sets  of  general-purpose  registers,  including  Stack  Pointer,  and  256/512  words  of  main 
memory.  It  can  simulate  computer  internal  behaviour  in  a register-transfer  level.  For  example,  a teacher  of 
Computer  Architecture  can  utilize  it  to  demonstrate  graphical  view  about  von  Neumann  computer 
architecture  by  means  of  wall-hanging  screen  and  PC  projector.  The  teacher  can  explain  step-by-step  actions 
of  internal  register  and  memory  with  VisuSim.  Fig.l  shows  an  overview  of  VisuSim  which  is  downloaded 
from  a server  and  executes  on  PC. 


Figure  1.  VisuSim  on  PC 


Figure  2.  Communication  between  pair  of  VisuSim  with  buit-in  e-mail  functions 
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2.2  e-Learning  Tool  with  Communication  Facility 

VisuSim  is  utilized  for  writing  essays  and  answering  problems  of  Assembly  Programming  exercise.  Users 
(Students)  of  VisuSim  can  understand  computer  behaviour,  program  processing  and  well-defined  algorithm 
through  verification  of  assembly  program  execution.  And  moreover  communication  service  can  be  provided 
between  pair  of  VisuSim  with  its  built-in  e-mail  function.  So  it  supports  effective  information  sharing 
between  sender  and  receiver  of  VisuSim.  It  is  very  useful  in  an  e-Learning  tool  among  users  in  Fig.2. 


3.  EVALUATION  OF  VISUAL  COMPUTER  SIMULATOR 

This  section  describes  an  evaluation  of  the  visual  computer  simulator.  It  presents  the  detail  of  essay  problems 
in  the  lecture,  analyzes  the  scores  of  students’  essays,  short  test  and  semester-end  examination.  And  then  it 
shows  the  relations  of  them  as  an  evaluation  of  VisuSim  in  the  relevant  year.  And  it  also  explains  qualitative 
and  quantitative  evaluation  in  the  second  half. 

3.1  Essay  Problems  for  Computer  Architecture  Education 

It  is  important  for  students  to  learn  vonNeumann  computer  architecture  as  efficiently  as  possible,  because 
such  a theme  is  positioned  as  an  introductory  talk  of  lecture  and  its  perfect  understanding  makes  more  fruitful 
and  expansible  for  the  succeeding  education  in  faculty  or  graduate  school  of  university.  Just  like,  for 
example,  understanding  Pipeline  Processing  as  higher  performance  technology  is  a introductory  talk  for  the 
advanced  architecture  themes  such  as  Super -pipeline.  Superscalar,  VLIW  and  so  on.  In  our  university,  a 
semester  has  only  15  weeks  for  every  lecture  and  always  suffers  from  lack  of  time  enough  to  educate  several 
items  for  students.  This  is  one  of  reasons  to  develop  our  visual  computer  simulator  described  in  the  previous 
section.  From  such  a viewpoint  for  lecture,  it  is  necessary  to  utilize  an  educational  tool  for  efficient  lecture 
and  provide  an  e-Learning  tool  for  effective  exercise  environment.  Essay  problems  of  Computer  Architecture 
are  shown  in  the  following  style: 

10  3 

(1)  2^(2*i)  + 10  , (2)  ^A(0^A(0)-1000,A(1)  = 900,A(2)-50,A(3)  = 5 

/=1  1=0 

These  are  useful  to  understand  Arithmetic  Algorithm  with  iteration,  register-register  operation,  and  usage 
of  index-register  based  indirect  address  modification.  If  a student  writes  such  programs  and  investigate  them 
using  our  visual  simulator,  he/she  can  recognize  simple  concept  of  vonNeumann  computer  architecture. 

3.2  Relation  between  Essay  and  Two  Kinds  of  Tests  for  Computer  Architecture 

In  lecture  of  "Computer  Architecture",  students  are  requested  to  write  assembly  programs  and  verify  their 
programs  with  our  visual  simulator.  At  the  same  time,  they  also  ought  to  check  by  themselves  whether  they 
understand  vonNeumann  computer  architecture  or  not.  Contents  of  a short  test  include  some  items  other  than 
assembly  programming  or  algorithm.  So  there  is  not  always  sure  that  the  students,  who  used  to  operate  our 
visual  simulator  very  well,  obtain  excellent  scores  in  such  a test.  It  can  be  assumed  that  the  contents  of  the 
test  have  very  few  factors  to  relate  skill  of  visual  simulator.  Under  these  conditions,  there  is  a result  of 
comparison  between  Essay's  Scores  from  students  and  Scores  of  Short  Test  by  the  same  students  shown  in 
Fig.  3. 

There  is  a certain  correlation  of  Scores  between  Essay  and  Short  Test.  So  we  can  confirm  that 
understanding  vonNeumann  computer  architecture  is  significantly  affected  by  writing  essays  with  usage  of 
visual  computer  simulator,  VisuSim. 

It  is  also  investigated  whether  there  is  any  relation  of  Scores  between  Essay  and  semester-end 
Examination  or  not.  It  has  been  not  yet  a perfect  investigation  and  confirmation  to  analyze  any  relation 
between  Essay  and  Examination  yet.  We  should  not  exactly  mention  about  a kind  of  relation  between  Essay 
and  Examination.  So  we  only  show  some  kind  of  good  correlation  of  Scores  between  Essay  and  Examination 
as  is  shown  in  Eig.  4. 
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Figure  3.  Relation  of  Scores  between  Essay 
and  short  Test  of  the  same  student. 
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iicores  of  semester  Hxamlnatlon 

Figure  4.  Relation  of  Scores  between  Essay 
and  semester-end  Examination. 


3.3  Qualitative  and  Quantitative  Evaluation  of  our  Simulator 

There  are  some  comments  and  recommendation  described  in  Essays  from  the  students,  who  learned 
"Computer  Architecture"  in  2009.  We  think  that  these  comments  and  others  scripts  are  considered  to  be  real 
and  precious  evaluation  for  our  simulator.  They  must  be  subjective,  of  course.  However,  they  can  be  also 
qualitative  evaluation  about  our  simulator  VisuSim  from  the  students  who  were  real  users  of  it.  Some  of 
positive  comments  are  as  follows: 

1)  It  is  easy  to  verify  and  point  out  mistakes  through  step-by-step  execution  and  display  the  result  by 
VisuSim. 

2)  I can  check  change  for  contents  of  registers  and  memory  provided  by  VisuSim. 

3)  As  I use  sample  programs  through  VisuSim,  it  is  good  enough  for  me  to  write  new  similar  ones. 

4)  It  is  very  important  that  I can  easily  understand  how  a computer  works,  because  VisuSim  visualizes 
interpretation  and  calculation  of  my  programs  graphically. 

5)  I can  recognize  characteristics  of  Stored  Programming  during  writing  programs  and  my  essay. 

The  above  positive  comments  may  be  considered  as  positive  evaluations  for  our  visual  simulator.  At  the 
same  time,  however,  there  are  some  of  negative  comments  in  the  relevant  Essays,  which  are  as  follows: 

6)  It  is  difficult  for  me  to  operate  VisuSim,  so  I must  help  my  friends  teach  me  to  operate  it.  Einally  I do 
write  my  essay  only  just  until  deadline. 

7)  When  I modify  my  program,  I have  found  mistake  at  the  point  where  program  jump  from  another  routine. 
So  I must  adjust  and  shift  the  according  routine. 

These  are  not  convenient  so  that  I think  it  is  necessary  to  be  improved.  These  comments  must  be 
considered  to  be  precious  recommendations  for  improvement  of  VisuSim  and  teaching  methods.  We  should 
have  a plan  to  modify  our  visual  simulator  from  these  feedbacks 

"Computer  Literacy"  is  an  introductory  lecture  for  beginners  of  "Computer  Architecture"  at  the  first  year 
of  our  university.  Table  1 shows  the  relation  of  learners’  scores  between  the  reports  using  VisuSim  for 
understanding  computer  and  examinations  for  "Computer  Literacy".  The  number  of  learners  is  81.  The  ranks 
of  the  scores  for  reports  are  classified  from  Ah-  to  D (ascending  order),  while  the  ranks  of  the  scores  for 
examination  are  "Superior",  "Excellent",  "Good",  "Eair"  and  "No  Good",  respectively.  Based  on  this  table, 
we  try  to  evaluate  our  visual  simulator  VisuSim  by  means  of  test  of  significance  as  follows. 
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Table  1.  Distribution  of  Learners'  Scores  for  Report  and  Examination 


Sunerior 

Excellent 

Good 

Eair 

NoGood 

number 

A-h 

8 

13 

3 

2 

1 

27 

A 

2 

15 

10 

4 

0 

31 

B 

0 

0 

3 

2 

0 

5 

C 

0 

0 

1 

2 

5 

8 

D 

0 

0 

0 

2 

8 

10 

total 

10 

26 

17 

12 

14 

81 

HO  is  a null  hypothesis:  "using  VisuSim  is  independent  from  learners'  scores  of  Computer  Literacy."  A 
chi-square  test  is  used  to  decide  whether  the  null  hypothesis  HO  holds  or  not.  With  data  of  Table  1,  the  chi- 
square  statistic  is 

2_(8-27-10/8lf  , , (8-10-14/81)" 

y — h • — I — o/.j  . 

27-10/81  10-14/81 

By  the  way,  the  significance  of  the  chi-square  statistic  is  evaluated  for  (5-l)*(5-l)  degrees  of  freedom. 
The  chi-square  percentile  with  (5-l)*(5-l)  degrees  of  freedom  at  the  5%  significance  level  is 

;To".o3(16)  = 26.3  . 

From  this  result,  the  null  hypothesis  HO  is  rejected  and  the  alternative  is  accepted.  An  evidence  of  the 
above  analysis  indicates  that  a relation  between  using  VisuSim  to  understand  "Computer  Literacy"  and  the 
learners'  scores  of  "Computer  Literacy"  shows  a statistically  significant  dependency.  So  it  is  said  to  be 
statistically  significant  that  using  of  our  visual  simulator  is  effective  to  learn  "Computer  Literacy". 


4.  COLLABORATIVE  LEARNING  IN  MULTIPLE  SERVER  SYSTEM 
ENVIRONMENT 

This  section  explains  two  types  of  applications  of  distributed  multiple  server  system,  namely,  implementation 
of  collaborative  learning  environment  and  distributed  surveillance  system  for  emergency  communication. 

4.1  Collaborative  Learning  Environment 

In  order  to  manage  our  educational  tool  effectively,  it  is  indispensable  to  design  and  implement  a special- 
purpose  information  server,  which  can  provide  some  kinds  of  information -exchanging  environment  for  the 
tool  and  its  users.  With  such  a server,  the  tool  can  play  the  very  important  role  to  carry  out  communication 
among  users.  Built-in  e-mail  handlers  of  the  tool  realize  such  communication  between  users,  i.e.  a learner  and 
an  instructor.  Learners  using  our  tool,  visual  computer  simulator,  can  obtain  their  necessary  information  from 
the  special-purpose  information  server  through  its  communication  supporting  functions  according  to  their 
understanding  levels.  So  the  information  server  needs  the  following  three  basic  functions. 

4.1.1  Web  Service  Functions 

They  are  very  much  essential  to  deliver  the  program  (executable)  code  of  Simulator  and  sample  (source) 
programs  for  Simulator.  They  correspond  to  HTTP -based  communication  with  3-way  hand-shaking 
procedure.  Additionally,  They  support  FTP-based  data  transferring  service. 

4.1.2  e-Mail  Service  Functions 

Simulator  can  support  the  communication  and  information-sharing  mechanism  among  users  by  means  of 
SMTP-based  and/or  POP3-based  facilities.  It  is  necessary  for  the  server  to  be  implemented  to  provide  SMTP- 
transferring  server  function  and  POP3 -receiving  server  ones. 
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4.1.3  User  Management  Service  Functions 

There  must  be  user  management  functions  in  the  server,  not  only  because  of  POPS -service  but  also  because 
of  user  identification  to  recognize  user's  understanding  level.  The  former  is  necessary  to  realize  POP3- 
service,  while  the  latter  is  essential  to  specify  user's  level  to  utilize  simulator  more  effectively. 

With  these  functions,  the  special-purpose  information  server  prepares  necessary  and  minimal  conditions 
to  realize  communication  supporting  and  information-exchanging  environment  for  the  educational  tool  and 
its  users. 


4.2  Cooperation  among  Multiple  Servers  System 


The  previous  information  server  simply  employed  one  server  system  for  all  the  users'  management,  user 
identification  with  serial  number,  so  that  there  were  some  regulations  such  as  not  so  good  two-way 
authentication  among  users,  not  so  efficient  information  transmission  and/or  sharing  between  different  level 
of  users,  etc. 

Now  we  have  employed  a new  resolution  of  Cloud  Service  to  group  all  the  users  into  cluster  and  assign 
such  a cluster  to  one  server  as  one  of  temporary  expedients  for  management  of  users  among  multiple  servers. 
Of  course,  the  above  palliative  treatment  is  not  effective  for  cooperation  among  multiple  servers.  Fig.  5 
shows  implementation  of  collaborative  learning  environment  for  our  educational  tool  with  a distributed 
multiple  server  system. 


Figure  5.  Collaborative  learning  environment  on  multiple  server  system  for  Visual  Computer  Simulator 

It  is  necessary  to  establish  more  effective  methods  to  organize  and  cooperate  different  servers.  One  of 
those  is  to  utilize  e-mail  function  of  the  educational  tool  and  communication  facilities  between  its  users.  An 
then  all  the  users  of  the  tool  are  registered  on  the  shared  user  database  and  their  mail  spool  area  and  users' 
home  directories  are  created  in  the  shared  volume  area  by  means  of  NIS/NFS  or  SAMBA  (or  Network- 
Attached  Storage)  facilities. 

Flexible  user  identification  is  necessary  to  allow  hierarchical  user  naming.  With  introduction  of  LDAP 
(Lightweight  Directory  Access  Protocol)  based  authentication,  it  is  very  much  smooth  to  manage  the  users 
among  multiple  servers  and  easy  to  implement  flexible  user  authority  for  such  servers.  Although  this  is  a 
useful  method  for  user  management,  it  will  be  suffering  from  some  dangerous  intrusion  without  closed 
network  based  characteristics  and  benefits.  Security  problems  are  very  much  heavily  serious  and  expensive  to 
protect  correctly  and  urgently.  Additional  facilities  such  as  NAT/NAPT  (Network  Address  (&  Port) 
Translation)  mechanism  will  be  implemented  into  a new  information  server  simultaneously.  So  that  will  be 
one  of  future  problems  to  be  resolved  for  practical  cooperation  in  a distributed  environment. 
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4.3  Related  Works 

First  of  all,  visualization  is  an  absolutely  necessary  keyword  and  idea  to  improve  the  learner's  understanding 
level.  For  example,  when  instructors  educate  their  learners  about  computer,  they  want  to  use  effective 
educational  tools.  These  tools  are  expected  to  have  some  kind  of  function  to  visualize  what  is  difficult  to 
understand.  With  such  tools,  many  learners  will  understand  computer  in  a shorter  period  than  other  cases 
without  using  visualization  tool. 

4.3.1  Three  Simulator  Tools  for  Teaching  Computer  Architecture 

Yehezkel  et  al.  have  pointed  out  that  teaching  computer  architecture  is  not  an  easy  task.  So  they  provide  three 
types  of  simulators  with  visualizations  for  different  computer  architectures.  They  are  (a)  EasyCPU  for  the 
Intel  80x86  families;  (b)  Little  Man  Computer  for  a general  von  Neumann  computer  architecture;  and  (c) 
RTLSim  simulator  for  a MIPS-like  CPU  [1].  They  are  excellent  works,  but  their  GUIs  are  neither  general  nor 
common.  It  is  difficult  for  beginners  to  use  different  GUIs  of  education  tools  in  the  related  educational  fields. 

4.3.2  Simulators  over  the  Network 

Llamas-Nistal  et  al.  have  designed,  implemented  and  tested  a Web-based  learning  system  in  pure  Java.  They 
have  attempted  to  stay  within  those  standards  what  are  suggested  for  distance  learning,  particularly  Web- 
based  collaborative  distance  learning  [2].  From  their  paper,  an  architecture  education  of  their  system  seems  to 
be  well  conceived  because  of  its  visualization.  Their  academic  results  and  opinions  from  their  students  who 
have  utilized  their  system  are  generally.  But  the  function  of  simulation  is  limited  for  a mid-level  or  lower 
computer  course,  so  it  seems  to  be  not  so  useful  enough  to  be  applied  to  assembly  programming  exercise 
including  recursion. 

4.3.3  The  MARIE  Computer  Simulator 

Null  et  al.  have  prepared  MarieSim  (a  computer  architecture  simulator  based  on  the  MARIE  architecture)  to 
teach  beginners  to  study  computer  organization  and  architecture.  It  provides  interactive  tools  and  simulations 
to  help  them  deepen  their  understanding.  The  graphical  environment  for  MarieSim  is  written  in  Java  Swing 
and  seems  to  be  useful  in  introductory  computer  architecture[3].  But  MARIE  employs  accumulator -based 
simple  architecture,  so  that  it  is  not  so  suitable  enough  to  execute  recursive  assembly  program  and  moreover 
the  MarieSim  is  not  completely  web-based  but  Java  stand-alone  application.  It  is  not  so  convenient. 

Secondarily,  Web-based  educational  tools  have  realized  powerful  and  fruitful  results  from  scientific  field 
to  computing  one.  The  second  half  of  this  section  focuses  the  following  two  researches  as  the  related  works 
based  on  Web-based  e-Learning  system  with  effective  GUIs. 

4.3.4  Integrated  Component  Weh-hased  Interactive  Learning  Systems  for  Engineering 

Humar  et  al.  have  proposed  a strategic  approach  to  integrate  already-developed  components  for  development 
of  a web-based  learning  environment.  Although  examples  from  their  system  only  demonstrate  how  the 
system  can  be  used  with  a course  on  electromagnetism,  however,  their  basic  approach  must  be  applicable  in 
other  fields  of  engineering  and  natural  science  [4] . 

4.3.5  A Web-based  Educational  Environment  for  Teaching  the  Computer  Cache  Memory 

Grigoriadou  et  al.  have  introduced  a Web-based  educational  environment  for  teaching  the  computer  cache 
memory  and  shown  their  aims  to  support  and  enhance  the  learning  process  for  such  a special  and  normally- 
invisible  memory.  The  results  obtained  from  the  application/evaluation  of  such  a Web -based  environment  are 
to  indicate  that  the  simulation  and  such  an  approach  can  effectively  support  and  enhance  the  learning  process 

[5]. 

Other  related  works  are  listed  in  the  last  references,  as  follows;  Moure  et  al[6]  have  developed  The 
KScalar  simulator  not  only  for  "Computer  Architecture  Education"  but  also  for  normal  professional  usage. 
Dr.  Chen  [7]  has  designed  and  (maybe  as  his  trial  challenges)  implemented  an  Automated  Eeedback  System 
for  Computer  in  order  to  utilize  and  improve  his  and  other  teachers'  educational  environments.  Both  of  Huey- 
Ing  Lui  and  Min-Num  Yang  [8]  have  originally  designed  and  developed  an  excellent  e-Learning  system, 
which  can  provided  QoL  guaranteed  adaptation  and  personalization  in  e-Learning  systems.  Djordjevic  et  al. 
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[9]  have  already  developed  and  utilize  Web-based  Educational  System  for  Computer  Architecture  Teaching 
in  their  university.  And  finally,  we  would  like  to  introduce  our  related  work  about  VisuSim  in  paper  [10]. 

From  the  surveys  of  above  described  works,  we  have  decided  to  design  a Web-based  educational  tool 
with  visualization  and  implement  an  effective  GUI  in  order  to  support  user-friendly  learning  process.  The 
following  three  sections  illustrate  the  detail  of  our  Web-based  educational  tool  for  Computer  Literacy  and 
Architecture  as  focusing  the  characteristics  of  its  GUI. 


5.  CONCLUSION 

This  paper  describes  summaries  of  a visual  computer  simulator,  some  trial  evaluation  of  the  simulator,  and 
organization  of  multiple  server  system  for  collaborative  learning  with  the  simulator.  And  it  introduces  related 
works  for  computer  simulators  and  Web-based  e-Learning  tools.  Through  the  above  explanation,  our 
conclusion  is  summarized  as  follows: 

1)  The  visual  computer  simulator  has  been  evaluated  in  qualitative  and  quantitative  approaches  based  on 
users’  comments  and  statistical  analysis  for  relation  between  essay  and  two  kinds  of  tests  of  "Computer 
Architecture". 

2)  It  is  confirmed  to  be  statistically  significant  that  using  of  our  visual  computer  simulator  VisuSim  can  be 
effective  to  learn  "Computer  Literacy". 

3)  It  shows  an  overview  of  distributed  multiple  server  system  for  collaborative  learning  with  the  simulator. 
Such  an  environment  will  be  useful  and  effective  for  practical  education.  But  there  are  some  problems  to 
be  resolved  for  practical  education,  just  like  security  and  external  violation. 
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ABSTRACT 

The  use  of  multimedia  instructions  for  online  learning  has  become  very  common  particularly  with  the  advances  of  the 
Internet  technology.  Consequently  museums  around  the  world  utilize  such  information  and  communications  technology 
(ICT)  tools  in  order  to  provide  richer  learning  experiences  for  their  visitors.  This  paper  discusses  a study  that  investigated 
the  relationship  between  multimedia  instructional  formats  with  individual  cognitive  learning  preferences  in  a museum 
learning  environment.  A total  of  91  school  children  age  between  10  to  12  years  old  were  randomly  assigned  into 
treatment  groups  based  on  their  cognitive  learning  ratio.  We  employed  a pre-test  post-test  quasi  experimental  design  to 
reveal  that  general  performance  of  the  children  exposed  to  the  physical  museum  exhibits  is  better  than  the  online  museum 
environment.  Although  single  cognitive  learning  preferences  were  evaluated,  our  findings  suggest  that  analytics  perform 
better  than  the  wholists  when  exposed  to  the  physical  exhibits;  whilst  the  result  is  reversed  for  the  online  exhibits 
environment.  Verbalisers  were  found  to  be  better  than  visualisers  in  the  physical  a museum  context.  Yet  they  were  found 
to  have  slight  differences  when  compared  to  visualisers  in  an  online  environment.  Our  findings  on  the  combined 
cognitive  styles  (CCS)  show  that  the  analytics-visualisers’  mean  scores  were  different  between  physical  and  online 
exhibits,  compared  to  the  other  three  CCS. 


KEYWORDS 

Multimedia  instructions,  cognitive  preferences,  museum  learning,  web-based  learning. 


1.  INTRODUCTION 

Multimedia  is  simply  defined  as  the  use  of  text,  graphics,  animation,  audio  and  video  to  present  information. 
The  revolution  of  the  Internet  and  the  communication  technologies  has  foreseen  the  information  to  be 
delivered  or  made  available  in  computer-based  instruction  utilizing  multimedia.  Combination  of  these  media 
or  better  referred  as  multimedia  instructions  allows  information  to  presented  in  a better  way  as  compared  to 
information  dissemination  in  a single  format.  As  suggested  by  Mayer  (2009)  and  Schnotz  and  Lowe  (2003), 
learning  from  multiple  formats  of  instructions  presumptively  resulted  in  a better  learning  outcome  as  opposed 
to  learning  from  a single  format  instruction.  Nevertheless,  there  are  research  that  proven  otherwise  (for 
example:  Rasch  & Schnotz  2009),  hence  the  proposition  remain  inconclusive.  Additionally,  review  of 
literature  shows  that  in  order  to  gain  a more  holistic  understanding  about  how  multimedia  instruction  could 
support  learning,  some  other  factors  pertaining  to  the  learning  process  such  as  the  learning  environment, 
learners’  characteristics  as  well  as  institutional  and  administrative  aspects  should  be  considered  when 
investigating  the  effectiveness  of  such  learning  instructions  (Tallent-Runnels  et  al.  2006). 

Web-based  learning,  which  involves  ICT  multimedia  tools,  has  emerged  to  overtake  the  more  traditional 
forms  of  instructional  environments.  In  doing  so  however,  the  increased  adoption  of  this  alternative 
pedagogical  regimen  may  raise  questions  to  doubt  its  effectiveness,  such  as:  does  the  combination  of 
different  media  really  work  in  such  learning  environments?;  which  combination  works  and  with  whom?;  and 
many  other  questions  that  will  require  answeres.  Research  has  also  shown  that  not  every  instructional  format 
works  for  everyone.  Schnotz  (2008)  and  Kolloffel  (2012)  for  example,  suggest  that  effectiveness  of 
animation  when  used  as  an  instructional  format  depends  on  factors  like:  learners’  preferences  towards  verbal 
or  visual  in  format;  prior  domain  knowledge;  and  other  learners’  personal  learning  characteristics.  This 
dilemma  suggests  that  other  factors  may  be  playing  a part.  Therefore  learners’  characteristics  should  be 
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closely  taken  into  account  when  designing  multimedia  instruction  strategies.  Obviously,  it  requires 
concentrated  attention  that  are  caused  by  the  human-computer  interaction  (HCI).  The  diversity  of  learner 
characteristics  forces  careful  consideration  to  address  an  individual’s  specific  learning  requirements. 

Children  for  example,  have  their  own  perceptions  towards  receiving  their  learning  instruction  given  to 
them  through  multimedia.  Previous  research,  which  focused  on  children’s  performance  when  learning  from 
multimedia  instruction,  indicated  there  were  elevated  promises  afforded  by  such  ICT  tools.  However,  the 
findings  remain  inconclusive  (for  example  see:  Grimley  2007;  Silverman  & Hines  2009).  This  disparity  is 
due  to  the  other  factors  or  other  learner  characteristics,  which  should  be  taken  into  consideration,  when 
designing  for  multimedia  learning  environments.  Herewith,  we  are  suggesting  that  investigating  the  way 
people  process  their  information  may  provide  a deeper  understanding  of  the  effectiveness  of  ICT  multimedia 
tools  which  adopt  such  instructional  pedagogies  in  helping  the  instructional/learning  process.  Individual 
cognitive  learning  preferences,  as  such  depict  which  is  their  preferred  and  habitual  approach  to  organize  and 
represent  the  instructional  information  they  receive.  This  characteristic  potentially  provides  “an  extensive 
and  more  functional  characterization  of  students’’  (Messick  1984).  Cognitive  style  is  a human  psychological 
dimension  that  is  integrally  linked  to  a person’s  cognitive  system  (Peterson,  Rayner  & Armstrong  2009), 
whereby  it  is  unique  and  likely  to  be  a fixed  aspect  of  a person  cognitive  functioning  (Riding  & Rayner 
1998). 

In  the  cognitive  psychology  field,  several  cognitive  style  models  have  been  developed.  However,  it 
should  be  noted  that  they  have  been  derived  from  researchers’  perspectives  and  varied  contexts.  Hence  in 
attempting  to  settle  on  one  definition,  it  widens  the  scope  and  becomes  very  confusing.  Riding  and  Cheema 
(1991)  consolidated  and  categorized  these  various  cognitive  styles  models  into  two  dimensions;  the  wholist- 
analytic  and  verbalizer-visualizer  dimensions.  The  wholist-analytic  dimension  describes  the  way  an 
individual  processes  the  information  they  receive,  whilst  the  verbal-visual  dimension  explains  the 
information  representation  strategy  an  individual  adopts  as  they  think  about  the  information  they  receive 
(Riding  & Sadler-Smith  1997). 

The  cognitive  styles  model  suggested  by  Riding  has  given  rise  to  a computerized  testing  tool  called  the 
Cognitive  Styles  Analysis  (CSA)  system.  The  CSA  is  used  to  assess  a person’s  position  on  the  continuum  for 
both  wholist-analytic  and  verbal-visual  dimensions,  based  on  the  computer-generated  ratio.  Despite  the 
critics  on  its  reliability  (Peterson,  Deary  & Austin  2003),  the  CSA  remains  to  be  one  of  the  most  systematic 
and  useful  tools  to  identify  a person’s  cognitive  (learning)  preferences.  The  CSA  has  been  used  by 
researchers  for  many  years.  Based  on  the  two  (Riding)  dimensions  of  cognitive  style,  a person’s  cognitive 
preference  is  anticipated  to  be  one  of  four  style  groups:  analytic -verbaliser;  analytic-visualiser;  wholist- 
verbaliser;  or  wholist-visualiser.  Each  of  the  four  style  groups  may  have  different  basic  preferences  towards 
mode  of  instruction.  Based  on  this  catagorization,  a learner  from  the  analytic-verbaliser  category  may  prefer 
text  in  contrast  to  those  analytic-visualisers,  who  may  perform  better  given  a captioned  picture  or  diagram. 
Therefore,  it  is  likely  that  different  individuals  with  different  cognitive  learning  preferences  will  perform  an 
other  way  in  another  instructional  context. 

Based  on  the  arguments  discussed  above,  this  research  was  conducted  to  investigate  the  interplay  between 
multimedia  instructions  offered  in  a web-based  museum  learning  environment  and  the  learners’  cognitive 
style  preferences.  This  paper  provides  the  context  of  investigation  as  well  as  explaining  the  experimental 
design.  The  findings  will  be  discussed  based  on  the  results  of  the  single  cognitive  preference  dimension 
(CPD)  and  the  combination  of  the  full  Riding  cognitive  style  dimensions  (CCS). 


2.  CONTEXT  OF  INVESTIGATION 

Many  museums  around  the  world  have  now  ventured  into  the  web-based  environment.  Consequently,  it  is 
pertinent  to  investigate  the  effectiveness  of  multimedia  ICT  tools  that  are  being  used  to  provide  richer 
museum  learning  experiences  for  their  visitors.  This  investigation  foreshadows  an  innovative  transformation 
of  the  instructional/learning  environment;  in  conjunction  with  the  more  traditional  role  a physical  museum 
plays  in  facilitating  a formal  learning  experience.  Furthermore,  acceptance  of  web-based  museums  as  being  a 
medium  of  communication  and  shared  information  is  now  seen  more  as  contributing  to  acknowledgement 
that  museum  institution  supports  such  transformation.  To  this  end,  museums  are  taking  the  advantages 
afforded  by  the  web-mediated  ICT  tools  to  enhance  their  communication  and  educative  interactions.  These 
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digital  pedagogies  support  the  physical  museum  experience,  thus  creating  a new  dimension  for  their  visitors' 
affective  experiences.  The  museum’s  role  as  a communication  medium  has  long  been  recognized  (Lord 
2007).  Hence,  the  use  of  multimedia  instructions  in  the  web -based  museum  is  intended  to  boost  the 
information  delivery  whilst  foster  learning  (Mayers  2009).  However,  many  suggest  that  even  professionally 
developed  instructions  have  failed  to  achieve  recognizable  learning  benefits  (Spector  and  Davidsen  2000; 
Schnotz  and  Lowe  2003).  This  heightened  museum’s  communication  role  cannot  be  seen  only  as  the  process 
of  delivering  information;  it  also  delivers  knowledge  acquisition  opportunities  to  suit  the  museum’s 
educational  aims  and  objectives.  The  opportunities  offered  by  the  web-based  environment  may  provide  a 
wider  (cognitive)  space  for  online  visitors  in  their  information  processing  (or  meaning  making). 

As  a result  it  is  important  to  note  here  that  it  is  the  way  information  is  represented  influences  how 
individuals  attend  to  appropriate  pieces  of  information  (Kolloffel  et  al.  2009;  Mendelson  & Thorson  2004) 
that  challenges  both  learners  and  designers.  The  fact  that  learning  is  highly  influenced  by  the  environment  in 
which  learning  takes  place  (Gagne,  (1985)  has  been  reflected  in  the  contextual  model  of  museum  learning 
(Falk  & Dierking  1992;  2000),  The  situation  in  which  the  role  of  context  is  emphasized,  contemplates  that 
learning  is  a process  occurring  under  certain  instructional  conditions,  with  the  effects  likely  to  vary  among 
individuals  (Gagne  1985).  However,  it  is  important  to  differentiate  between  learning  and  instruction,  as 
learning  theories  explain  what  happens  in  the  learner’s  head  whilst  instructional  theories  describe  the 
conditions  which  facilitate  learning  (Reigeluth  1983).  There  are  extensive  changes  in  the  supporting 
technologies  available,  whilst  enriching  the  museum  learning  experience.  It  could  be  argued  that  the  potential 
of  such  technologies  may  only  be  realized  if  the  design  and  use  of  the  ICT  tools  are  designed  and  developed 
drawing  upon  our  understandings  of  how  the  users  learn  Laurillard  (2006). 

It  is  with  this  background  that  this  research  project  was  conducted  on  an  exploration  of  students’  learning 
experiences  in  an  online  museum  environment.  The  research  aim  was  to  investigate  the  relationship  between 
the  museum’s  ICT  enhanced  (multimedia)  instructional  strategies  with  students’  cognitive  learning 
preferences  in  their  museum  learning  performance.  There  is  still  much  to  be  discovered  about  how  learners 
interact  with  multimedia  enhanced  instructional  strategies  online.  Exploring  the  learners’  individual  cognitive 
preferences  provides  an  insight  of  their  working  memory  during  the  learning  process  hence  afford  a better 
understanding  of  the  relationship  between  the  two  factors.  This  increased  understanding  will  assist  to  identify 
the  instructional  conditions,  which  enable  and  facilitate,  rather  than  hinder  the  learning  process. 


3.  THE  EXPERIMENTAL  DESIGN 

There  were  91  schoolchildren  age  10  to  12  years  old  chosen  to  participate  in  this  research.  They  visited  the 
museum  (brought  there  on  a special  but,  by  their  teachers  as  an  educational  field  trip).  They  were  to  learn 
from  a certain  exhibit  that  related  to  what  they  would  be  expected  to  learn  in  their  classroom  setting.  This 
group  of  children  were  recruited  using  a convenient  sampling  technique  from  three  schools.  Their  prior 
knowledge  was  considered  in  the  experiment’s  design.  They  were  anticipated  to  share  similar  familial 
backgrounds  and  to  have  received  the  same  level  of  educational  experience  as  others  of  the  same  group. 

The  fieldwork-experiments  employed  a three-phase  quasi-experimental  design.  The  first  phase  involved  a 
screening  test  to  measure  the  participants’  cognitive  preferences,  using  the  CSA  (Riding  1991)  screening  test. 
The  CSA  and  a pre-test  were  conducted  prior  to  the  children’s  museum  visit.  The  purpose  was  to  determine 
their  prior  domain  knowledge  as  it  related  to  the  forthcoming  museum  exhibits.  Based  on  the  cognitive 
preferences  identified  from  the  CSA,  the  participants  were  assigned  into  the  treatment  groups;  the  children 
were  given  either  the  web-based  museum  instruction  (Tl)  or  the  physical  museum  visit  as  their  treatment 
(T2).  Despite  the  distinctively  different  environments  for  Tl  and  T2,  the  multimedia  instructions  were  the 
same  (besides  some  other  parts  of  the  web-sites  that  offered  instruction  in  other  instructional  formats  such  as 
video).  Due  to  that  disparity,  for  the  purpose  of  this  experiment,  the  Tl  participants’  access  to  the  online 
exhibits  was  limited  to  the  digital  (computerized)  instructional  parts  that  replicated  the  instruction  in  the 
physical  museum  only  treatment  (see  explanation  in  Figure  2). 
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In  the  second  phase,  each  treatment  group  was  given  access  to  the  online  museum  or  the  physical 
museum  treatment  respectively.  For  the  online  session,  30  minutes  was  allocated  to  the  participants  to  browse 
the  existing  web  pages  of  the  Dinosaur  Walk  exhibition  in  the  Melbourne  Museum  website  (Figure  1). 
Meanwhile,  participants  of  the  physical  visit  treatment  group  were  taken  to  physically  explore  the  Dinosaur 
Walk  exhibition  (for  example,  see  Figure  2 and  Figure  3)  in  the  Melbourne  Museum  within  the  same  length 
of  time.  The  experiment  was  concluded  with  a post-test  given  to  the  children  immediately  after  receiving 
their  T1  and  T2,  to  measure  any  improvement  in  the  cognitive  performance  (or  learning  outcomes)  derived 
from  their  museum  visit  as  learning  experience. 


Ammi^msmunjs  cmz»ui 


Figure  1.  A page  consists  of  the  information  of  a dinosaur  (courtesy  of  Melbourne  Museum). 


Figure  2.  Example  of  exhibits  display  in  the  physical  museum  exhibition. 


Figure  3.  Dinosaurs’  skeleton  on  display  at  the  physical  exhibition. 
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4.  ANALYSIS  AND  RESULTS 

The  data  gathered  from  the  fieldwork  experiment  was  then  analysed  using  the  Winstep  Software  that  applies 
Rasch  Measurement  Model.  The  model  is  probabilistic  and  inferential  therefore  allows  analysis  of  an 
individual  performance  relative  to  the  instrumentation  as  ""the  person  ability  and  item  difficulty  are  conjointly 
estimated  and  placed  on  a numerical  scale”  (Sick  2008)  called  logit.  A logit  is  a unit  of  measurement 
described  as  “interval  scale  in  which  the  unit  intervals  between  locations  on  person-item  map  have  a 
consistent  value  or  meaning”  (Bond  and  Fox  2007)  or  referred  as  uni-dimensionality.  This  occurs  when  the 
data  fit  the  model  and  reliability  of  item  placement  is  established. 

The  data  analysis  discussed  in  this  paper  was  conducted  by  looking  at  the  single  CPD  values  that 
differentiated  between  the  wholist-analytic  and  verbaliser-imager  dimensions;  extending  to  the  combination 
of  the  CCS  described  earlier  as:  wholist-verbaliser;  wholist-visualiser;  analytical-verbaliser  and  analytical- 
visualiser.  The  analysis  of  mean  score  intends  to  identify  the  relationship  between  the  cognitive  styles  and  the 
multimedia  instructions  with  students’  learning  performances  for  each  treatment  respectively.  To  do  this,  the 
mean  score  for  the  specific  CPD  and  CCS  was  compared  within  and  between  treatment  groups  in  order  to 
identify  the  relationship. 

In  general,  the  mean  analysis  indicated  that  the  overall  performance  of  participants  in  T2  were  better,  as 
compared  to  those  in  T1  for  both  CPD  and  CCS.  For  CPD,  the  result  yields  a mean  score  of  43.0  for  T2  over 
T1  with  mean  score  of  39.1.  When  comparing  general  performance  of  each  CPD  in  between  treatments,  the 
result  was  consistent  with  the  earlier  analysis  which  indicated  that  each  CPD  (wholist,  analytics,  verbaliser 
and  visualiser)  demonstrated  a better  performance  with  the  T2.  Further  analysis  of  CPD  in  T1  revealed  that 
the  verbalisers  recorded  the  highest  mean  score  of  39.9.  Whilst  the  wholist  mean  score  was  slightly  lower  at 
39.5.  Analytics  and  visualisers  were  found  to  be  at  par  in  their  performance  with  a mean  score  of  38.6  and 
38.5  respectively.  Analysing  the  result  of  CPD  in  T2  shows  that  all  CPD  had  higher  mean  scores  in  T2. 
Nevertheless,  the  highest  score  was  obtained  by  analytics  with  mean  score  of  44.8.  Verbalisers  that 
previously  scored  the  highest  in  T1  were  revealed  to  have  the  least  mean  score  at  42.6.  Overall  analysis  of 
CPD  showed  that  analytics  performed  better  than  wholists  in  T2;  yet  the  results  was  vice  versa  with  Tl.  As 
for  the  verbaliser-visualiser  dimension,  the  verbalisers  achieved  better  scores  with  the  Tl,  compared  to  the 
visualisers.  However,  both  the  verbalisers  and  visualisers’  mean  scores  were  found  slightly  differ  with  the 
T2.  This  analysis  is  simplified  in  Figure  4. 


Further  analysis  was  conducted  for  CCS.  In  general  it  resulted  showing  that  the  analytic-verbalisers  and 
wholist-visualiser  achieved  leveled  performances  with  mean  scores  of  41.8  and  41.7  respectively.  Whilst  the 
wholist-verbaliser  mean  score  was  40.9;  the  analytic-visualiser  scored  the  least  at  37.3.  Analysis  of  CCS  for 
each  treatment  provided  further  detail  of  their  performances.  For  Tl,  the  analytics-visualisers  were  found  to 
score  the  least  with  a mean  score  of  35.7;  whilst  the  analytics-verbalisers  demonstrated  that  their  highest 
ability  was  to  score  at  the  40.4  level.  The  wholist-verbaliser  and  wholist-visualiser  performed  equally  well 
with  mean  score  differences  of  only  0.2.  The  mean  scores  analysis  in  T2  revealed  that  the  analytics- 


29 


ISBN:  978-972-8939-88-5  © 2013  lADIS 


verbalisers  group  remained  as  the  top  scorers.  On  the  other  hand,  wholist-visualisers  performances  were 
better  compared  to  the  wholist- verbalisers  with  mean  score  differences  of  1.6.  Interestingly,  analytics- 
visualisers  recorded  a difference  of  7.3  when  compared  with  the  mean  scores  in  Tl.  The  overall  result  for  the 
CCS  is  depicted  in  Figure  5. 


Figure  5.  Mean  score  analysis  between  CCS  according  to  treatment 


5.  FINDINGS  AND  DISCUSSION 

The  aim  of  this  paper  aims  was  to  explore  the  effects  of  cognitive  learning  preferences;  namely  the 
verbaliser-visualiser  and  wholist-analytics  dimensions  on  children’s  museum  learning  experiences.  The 
research  was  conducted  by  focusing  on  the  web-based  museum  exhibits  whilst  comparing  them  with  the 
physical  museum  exhibit  for  the  Dinosaur  Walk  exhibit.  The  data  analysis  was  conducted  in  two  stages.  The 
first  was  by  comparing  between  single  CPD  and  within  the  CCS.  For  the  analysis  of  CPD,  the  results 
demonstrate  that  wholists’  performance  was  better  than  the  analytics  with  Tl  (the  web-based  museum  exhibit 
environment).  However,  the  results  were  vice  versa  with  the  T2;  whereby  the  analytics  achieved  higher  mean 
as  compared  with  the  wholists.  The  same  pattern  was  also  observed  for  the  verbaliser-imagery  dimension, 
where  the  imagers’  performances  increased  with  T2.  Whereas,  the  verbalisers  showed  a decline  with  their  T2 
mean  score.  Further  analysis  on  the  means  of  both  CPD  revealed  that  both  dimensions  had  an  interaction  with 
the  different  instructional  museum  exhibit  formats;  therefore  suggesting  that  cognitive  learning  preferences 
do  have  an  effect  on  museum  learning  performance. 

On  the  other  hand,  the  CCS  analysis  revealed  that  the  wholists  with  preferences  in  either  the  verbal  or 
imagery  dimension,  have  similar  performances  with  only  slight  differences  in  their  mean  scores.  This  result 
suggests  that  the  wholists,  whom  presumably  process  information  they  receive  as  a whole  (Riding  and 
Cheema,  1991),  benefited  from  the  combination  of  text  and  graphical  information  in  the  web-based  museum 
environment;  despite  their  verbal  or  visual  learning  preferences.  However,  there  was  a significant  difference 
for  the  analytics.  From  these  results,  it  was  shown  that  the  analytics  with  verbal  preferences  performed  the 
best,  whilst  the  analytics  with  a visual  preference  performed  the  worst.  This  result  indicated  that  the  analytics 
outperformed  the  wholists  in  their  web-based  museum  learning  performances  when  they  were  a verbaliser. 

The  way  information  is  presented  in  a physical  museum  (scattered  individually  as  objects  or  individual 
exhibits)  allows  analytics  to  process  the  information  in  chunks  (Riding  and  Cheema,  1991).  Hence  they 
perform  better  than  the  wholists.  Whereby,  the  combination  of  both  textual  and  graphical  information 
received  in  a web-based  museum  gave  more  advantages  to  children  who  were  verbalisers  than  the  visualisers. 
As  suggested  by  Kim  and  Gilman  (2008),  reduced  amounts  of  screen-based  text  provide  ample  space  for 
graphical  instruction  that  assists  certain  learners  to  understand  better.  However,  combination  of  both  textual 
and  graphical  instructions  displayed  together  in  a web-based  museum  exhibit  may  also  distort  the  focus  and 
concentration  of  the  visualisers;  conforming  to  the  findings  by  Schnotz,  Banner!  and  Seufert  (2002).  Besides, 
as  some  of  the  information  was  displayed  in  either  text  or  graphical  format  only;  this  may  possibly  cause  the 
visualisers  to  focus  more  on  the  images  and  miss  some  of  the  verbal  information. 
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Both  the  CPD  and  CCS  data  analysis  demonstrates  that  the  visualisers  had  lower  performances  with  the 
T1  as  compared  to  the  verbalisers.  This  result  therefore  suggests  that  findings  from  this  research  are 
contradicting  the  previous  findings  of  Parkinson  and  Redmond  (2002),  and  Riding  and  Douglas  (1993),  when 
they  suggested  that  the  visualisers  should  perform  better  with  combination  of  text  and  graphics  in  a learning 
environment  when  compared  to  the  verbalisers.  However,  the  premise  remains  true  for  the  physical  museum 
exhibits. 


6.  CONCLUSION 

This  study  investigated  the  effectiveness  of  multimedia  instruction  strategies  that  contained  text  and  graphics, 
in  delivering  museum  information,  particularly  in  a web-based  exhibit  environment.  A comparison  was  made 
to  the  traditional  museum  exhibits  that  offer  similar  information  using  the  same  multimedia  instructional 
formats.  Obviously,  general  performances  of  the  participating  children  were  better  in  the  physical  museum 
context  when  compared  to  the  web-based  museum  environment.  This  finding  indicates  that  a web-based 
learning  space  is  useful;  yet  the  traditional  learning  context  remains  as  the  most  important.  Based  on  the 
results  discussed  earlier,  it  can  be  concluded  that  web-based  museum  exhibits  using  the  combination  of  both 
textual  and  graphical  information  benefits  both  wholists  and  analytics.  However,  the  nature  of  the  web-based 
information  representation  with  such  combination  may  provide  more  advantages  to  verbalisers  than 
visualisers.  Nevertheless,  this  paper  only  reports  the  comparison  between  the  instructional  strategies  of  the 
museum  exhibits  (web-based  and  physical  museum  exhibits).  For  future  work,  it  would  be  interesting  to 
explore  further  in  other  web -based  environments.  Factors  such  as  the  use  of  frame  or  information  structuring 
and  other  interface  design  issues  that  are  likely  to  interact  with  cognitive  learning  preferences  to  affect  the 
instmctionaPlearning  performance.  Additionally,  involving  users  during  the  design  and  development  or 
evaluation  process  of  such  learning  environment  may  also  provide  richer  information  and  detail 
understandings. 

Apart  from  that,  the  study  also  revealed  that  children  with  certain  cognitive  preference  may  perform 
differently  (in  physical  and  web-based  museum  environment).  This  result  has  resulted  from  the  way  they 
process  the  information  they  receive.  Whereby  a verbalizer  was  assumed  to  perform  better  when  given 
textual  instruction  and  imager  should  achieve  better  when  dealing  with  a graphical  instructional  strategy. 
However,  when  it  comes  to  multimedia  instructions  that  combine  both  text  and  graphics,  the  findings  could 
be  different  depending  on  the  context  of  investigation.  As  discussed  earlier,  this  study  demonstrated  that  the 
combination  of  text  and  graphics  gave  more  advantages  to  verbalizers  than  visualisers.  Looking  at  the 
wholists-analytics  dimension,  the  findings  revealed  that  the  wholists  preferred  the  physical  museum  exhibits, 
compared  with  a web-based  exhibit;  whilst  the  analytics  performed  better  in  the  web-based  museum  setting. 

The  overall  conclusion  that  is  drawn  here  is;  it  is  necessary  to  have  a good  understanding  of  the 
relationship  between  multimedia  instructional  formats  to  be  offered  (in  facilitating  the  learning  process)  with 
the  way  learners  perceive  the  instructional  information.  To  do  so,  an  investigation  may  be  essential  to  provide 
a deeper  understanding  of  the  learning  settings  as  each  is  unique  and  may  be  different  from  other  learning 
environment.  Furthermore,  those  findings  from  other  research  could  be  helpful  in  guiding  the  design  of  a 
learning  environment.  Yet  specific  consideration  of  that  particular  context  will  provide  more  explicit 
understandings  towards  designing  more  effective  instmctionaFlearning  strategies.  Furthermore,  this  study 
also  demonstrated  the  importance  of  allowing  for  learners’  cognitive  learning  preference  differences  when 
designing  multimedia  instruction.  This  means  to  cater  for  a broader  range  of  human  cognitive  abilities 
(McKay  2003). 
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ABSTRACT 

Homework  is  a daily  activity  for  at  least  twelve  years  of  most  students’  school  experience,  and  every  assignment  requires 
the  time,  energy,  and  emotional  engagement  of  all  those  involved.  Traditionally,  students  seeking  homework  help  could 
refer  to  their  class  notes  and  textbooks,  or  ask  their  friends,  tutors,  and,  perhaps,  as  last  resort,  their  parents.  Now, 
however,  the  Internet  has  greatly  extended  the  set  of  resources  to  which  students  have  ready  access.  By  going  online, 
students  can  read  tutorials,  watch  videos,  and  even  seek  personalized  homework  help  from  a large  community  of  others  in 
online  forums.  Students  who  are  currently  in  high  school  have  grown  up  with  computers,  mobile  devices,  and  other 
technologies  that  make  Internet  access  a convenience,  if  not  an  expectation.  Given  their  exposure  to  technology,  together 
with  an  expanded  pool  of  readily  available  resources,  how  do  students  today  seek  help  on  their  homework?  In  particular, 
what  resources  (digital  versus  non-digital)  do  they  favor  and  to  what  extent?  This  paper  documents  how  a large 
population  of  USA  high  school  students  seeks  help  on  their  mathematics  assignments.  Comparisons  between  students  in 
remedial,  core,  and  advanced  courses  are  also  made. 


KEYWORDS 

Digital  resources.  Help  seeking.  Homework,  Mathematics,  Net  Generation. 


1.  INTRODUCTION 

For  at  least  twelve  years,  homework  is  a daily  activity  in  most  students’  schooling  experience.  Every 
assignment  requires  the  time,  energy,  and  emotional  engagement  of  all  those  involved.  Traditionally,  students 
in  need  of  help  could  refer  to  their  class  notes  or  textbooks,  or  they  could  ask  their  teacher,  their  friends,  and, 
perhaps,  as  a last  resort,  their  parents  (Cosden,  Morrison,  Albanese,  &.  Macias,  2001;  Epstein  & van  Voorhis, 
2001;  Hoover-Dempsey  et  ah,  2001;  Solomon,  Warin,  & Lewis,  2002).  Now,  however,  the  Internet  has 
greatly  extended  the  set  of  resources  to  which  students  have  ready  access.  By  going  online,  students  can  read 
tutorials,  watch  videos,  and  even  seek  personalized  homework  help  from  a large  community  of  others  who 
take  part  in  online  forums.  Students  who  are  currently  in  U.S.  high  schools  have  grown  up  with  computers, 
mobile  devices,  and  other  technologies  that  make  Internet  access  a convenience,  if  not  an  expectation.  Given 
their  exposure  to  and  familiarity  with  technology,  together  with  an  expanded  pool  of  readily  available 
resources,  how  do  students  today  seek  help  on  their  homework?  In  particular,  what  resources,  digital  versus 
non-digital,  do  they  favor  and  to  what  extent? 

This  paper  addresses  these  questions  by  documenting  the  help-seeking  habits  of  high  school  students  on 
their  mathematics  assignments.  Drawing  together  themes  that  have  emerged  from  research  on  homework,  the 
‘Net  generation,’  and  digital  resources,  we  sought  to  answer  four  major  questions  through  a survey  of  more 
than  1800  students: 

1 What  resources  do  high  school  students  use  for  mathematics  homework  help,  and  with  what 
frequency?  a)  How  does  their  use  of  online  versus  offline  resources  compare?,  and  b)  Are  there 
differences  in  resource  usage  between  students  in  remedial,  core,  and  advanced  mathematics  courses? 

2 In  what  locales  do  high  school  students  complete  their  mathematics  homework? 

3 How  much  help  do  students  perceive  they  need  compared  with  what  they  currently  receive? 

4 How  is  desire  for  help  related  to  resource  use  and  awareness  of  existing  free  digital  resources? 
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1.1  Previous  Research 

A theoretical  framing  of  help  seeking  as  a legitimate  learning  activity  (Nelson-Le  Gall,  1985)  with  a broad 
impact  across  varied  educational  settings  sets  the  stage  for  our  study  which  hones  in  on  the  homework 
help-seeking  habits  of  today’s  high  school  students. 

1.1.1  Homework 

Homework  is  a shared  part  of  the  daily  school  routine  for  students  across  different  grade  levels,  generations, 
and  even  nationalities.  Because  homework  plays  such  a central  role  in  instruction,  it  has  been  an  active  area 
of  educational  research  for  almost  a century  (Cooper  & Valentine,  2001).  Although  the  findings  of  individual 
studies  from  within  this  corpus  are  often  conflicting,  syntheses  of  this  body  of  work  give  a good  sense  of  the 
relationship  between  homework  and  achievement,  attitudes  toward  homework,  and  homework  help  (Cooper, 
1989). 

One  of  the  major  purposes  of  homework  is  to  provide  students  with  additional  academic  practice  and 
learning  opportunities  outside  of  class.  Without  homework,  most  instructors  find  it  impossible  to  cover 
curriculum  content,  and  thus  view  it  as  an  essential  ingredient  for  learning  and  achievement  (de  Jong, 
Westerhof,  & Creemers,  2000).  Indeed,  in  a quantitative  synthesis  of  numerous  studies  comparing  homework 
with  no-homework,  homework  with  supervised  study,  and  measuring  differing  amounts  of  homework 
completed.  Cooper  (1989)  found  support  for  a positive  relationship  between  doing  homework  and 
achievement  with  few  exceptions.  Furthermore,  the  evidence  from  all  three  types  of  study  suggested  that  the 
relationship  between  homework  and  achievement  was  influenced  by  the  students’  age  or  grade  level.  Doing 
homework  was  more  strongly  associated  with  achievement  in  high  school  than  middle  school,  and  in  middle 
school  than  in  elementary  school.  Of  course,  any  positive  association  between  homework  and  achievement  is 
also  mediated  by  whether  or  not  students  receive  feedback  on  their  work  (de  Jong  et  ak,  2000)  and  by  the 
quality  of  the  assignments  (Epstein  & van  Voorhis,  2001). 

Given  the  increasing  role  homework  plays  in  achievement  over  the  course  of  schooling  regardless  of 
student  ability  (Keith,  1982;  Keith,  Reimers,  Fehrmann,  Pottebaum,  & Aubey,  1986),  it  is  unfortunate  that  on 
average  few  students  in  higher  grades  do  all  of  their  homework.  They  may  be  suffering  from  sheer  frustration 
(Anesko,  Schoiock,  Ramirez,  & Levine,  1987)  or  simply  not  ascribe  to  the  same  beliefs  about  the  benefits 
and  purposes  of  doing  homework  as  do  adults  (Warton,  2001).  In  particular,  students  do  not  view  homework 
as  a way  of  learning  valuable  time -management  or  study  skills,  even  though  parents  may  reduce  the  amount 
of  help  that  they  provide  in  the  expectation  that  their  children  will  become  progressively  more  autonomous 
and  personally  responsible.  In  addition  to  perceiving  their  support  role  as  becoming  more  emotional  rather 
than  academic  over  time,  parents  may  also  become  less  actively  involved  in  homework  because  of  feelings  of 
inadequacy  (Epstein  & van  Voorhis,  2001),  inadequate  information  (Hoover-Dempsey  et  ak,  2001),  and  the 
tension  that  often  results  when  they  become  involved  (Solomon  et  ak,  2002).  Some  of  the  slack  in  providing 
students  with  homework  help  has  been  taken  up  recently  by  after-school  programs  and  tutor  centers  (Cosden 
et  ak,  2001).  Here,  tutors  can  offer  effective  face-to-face  help  and  support  as  students  work  on  their 
homework  assignments. 

However,  technology  now  provides  the  opportunity  to  expand  help  seeking  support  from  such  small- 
scale,  local  tutoring  locales  to  much  larger,  geographically  dispersed  communities  and  learner  networks  (van 
de  Sande  & Leinhardt,  2007).  The  question  here  is  whether,  given  their  familiarity  with  and  exposure  to 
sophisticated  technology,  the  ability  to  relocate  homework  help  and  expand  available  resources  is  something 
that  students  today  value  and  of  which  they  take  advantage. 

1.1.2  The  Digital  Generation 

The  last  decade  has  seen  a massive  proliferation  of  increasingly  sophisticated  information  communication 
technology  (ICT).  This  boon  in  ICT  has  naturally  led  to  questions  and  concerns  regarding  the  fit  between  the 
educational  system  and  the  wants  and  needs  of  students  who  became  young  adults  at  the  time.  Although 
authors  differ  on  the  exact  date  brackets,  students  bom  roughly  between  the  early  1980’s  and  the  early  2000’s 
are  said  to  belong  to  the  “Net  generation”  (Tapscott,  1998,  1999,  2008)  and  are  characterized  as  “digital 
natives”  (M.  Prensky,  2001a,  2001b)  because  of  their  familiarity  with  and  reliance  on  ICT.  Other  authors  use 
the  term  “millenials”  to  emphasize  that  it  is  exposure  to  technology,  rather  than  age  alone,  that  characterizes 
developing  trends  in  how  students  learn  (Howe  & Strauss,  2000;  Oblinger  & Oblinger,  2005).  However, 
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whether  using  a definition  based  on  age  or  exposure,  proponents  for  the  ‘digital  movement’  hypothesize  that 
today’s  students  have  distinct  characteristics  that  set  them  apart  from  previous  generations  and  that  radically 
shape  how  they  learn.  The  claim  is  that  students  are  now  active  experiential  learners  who  are  facile  with 
multi-tasking  and  who  rely  on  communication  technologies  to  access  information  and  interact  with  others. 
Based  on  this  characterization  of  new  learning  styles,  commentators  warn  that  the  current  educational 
approach  is  ill  suited  for  the  task  of  helping  students  learn  and  must  be  overhauled  in  order  to  adequately 
serve  this  new  cohort  of  students:  “Our  students  have  changed  radically.  Today’s  students  are  no  longer  the 
people  our  educational  system  was  designed  to  teach”  [emphasis  in  original]  (Marc  Prensky,  2001,  p.  1).  The 
prediction  is  that  dire  consequences  will  result  if  there  is  not  a radical  change  in  how  students  are  taught  all 
subjects  at  all  levels. 

However,  not  all  researchers  agree  with  these  exhortations  for  full-scale  educational  transformation. 
Bennett,  Maton,  and  Kervin  (2008)  scrutinize  two  major  assumptions  underlying  the  claim  made  for  the 
existence  of  a generation  of  ‘digital  natives.’  Specifically,  they  question  whether  students  in  this  generation 
possess  truly  sophisticated  technological  skills  and  whether  they  have  different  learning  styles  compared  with 
previous  generations  of  learners.  Based  on  their  critical  review  of  the  research,  they  find  little  support  for 
these  assumptions.  The  conclusion  reached  is  that  the  outcry  for  the  necessity  of  radical  educational  reform  to 
meet  the  needs  of  the  ‘Net  generation’  instead  can  be  construed  as  a form  of  ‘moral  panic’  or  hype,  rather 
than  being  empirically  and  theoretically  informed.  In  its  place,  they  advocate  for  a more  thorough 
characterization  of  how  today’s  students  interact  with  ICT,  a goal  that  can  only  be  achieved  through 
“considered  and  rigorous  investigation  that  includes  the  perspectives  of  young  people  and  their  teachers,  and 
genuinely  seeks  to  understand  the  situation”  (p.  784).  Accordingly,  several  researchers  have  answered  this 
call  with  attempts  to  flesh  out  the  role  ICT  actually  plays  in  students’  lives,  and  specifically  with  regard  to 
ways  in  which  they  learn  (Helsper  & Enyon,  2009;  Jones,  Ramanau,  Cross,  & Healing,  2010). 

1.1.3  Digital  Resources 

How  might  students  of  the  Net  generation  use  ICT  to  get  help  on  their  homework  in  ways  that  differ  from  the 
help  available  to  previous  generations?  There  are  numerous  digital  resources  that  relate  to  homework  help, 
many  of  which  are  available  at  no  cost.  Students  who  have  questions  on  their  coursework  can  readily  find 
either  generic  help  (e.g.,  by  watching  an  instructional  tutorial)  or  more  personalized  help  (e.g.,  by  interacting 
with  others  on  an  online  forum).  By  typing  their  query  into  a search  engine,  students  can  almost 
instantaneously  get  a list  of  documents  or  webpages  that  might  contain  the  sought  after  answer,  or  they  can 
get  answers  to  factual  queries  directly  by  using  a sophisticated  computational  engine  (such  as  Wolfram 
Alpha).  These  are  just  a few  of  the  ways  in  which  ICT  has  made  it  possible  for  students  seeking  help  on  their 
homework  to  have  a wealth  of  information  and  communities  at  their  fingertips. 

However,  although  we  know  that  95%  of  students  ages  12  - 17  access  the  Internet  (Madden,  Lenhart, 
Duggan,  Cortesi,  & Gasser,  2013)  much  less  is  known  about  how  students  are  using  the  Internet  for 
homework  help.  For  instance,  although  Khan  Academy  (www.khanacademy.org)  has  published  more  than 
4,000  instructional  videos  covering  K-12  school  topics,  we  do  not  know  (other  than  anecdotally)  how 
watching  these  videos  translates  into  learning  or  resolving  student  questions.  With  regard  to  the  use  of  open 
online  homework  help  forums,  we  know  a little  more.  In  particular,  members  of  some  of  these  forums 
demonstrate  a strong  sense  of  virtual  community  in  that  they  share  responsibility  for  participation,  take  on 
different  roles,  and  work  to  create  a safe  place  for  people  to  share  their  thoughts  and  (lack  of)  understanding 
(van  de  Sande  & Leinhardt,  2007).  Furthermore,  because  forum  exchanges  are  archived  as  discussion 
threads,  it  is  possible  to  make  inferences  about  the  effect  of  the  interaction  on  student  understanding  (van  de 
Sande,  2010).  In  a sample  of  100  discussion  threads  on  the  topic  of  the  limit  from  one  mathematics  help 
fomm,  it  was  possible  to  infer  something  about  how  the  student  perceived  the  helpfulness  of  the  interaction 
in  36  cases;  the  student  demonstrated  increased  understanding  of  the  problem  solution  in  23  of  these  threads, 
expressed  gratitude  and  thankfulness  in  10  of  the  threads  (which  may  or  may  not  have  signaled  deeper 
understanding),  and  only  indicated  remaining  uncertainty  and  confusion  as  the  last  communication  in  3 of  the 
threads.  Naturally,  these  percentages  will  vary  depending  on  the  culture,  norms,  and  practices  of  the  site 
being  investigated  (van  de  Sande,  2010)  and  do  not  say  anything  about  long-term  learning  outcomes,  but 
such  studies  of  resolution  and  participation  do  contribute  to  our  understanding  of  how  students  who  belong  to 
the  ‘Net  generation’  are  using  at  least  some  types  of  digital  resources  for  seeking  homework  help. 
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2.  PARTICIPANTS  AND  METHODS 

Students  and  teachers  in  all  mathematics  classes  at  a large  public  high  school  (enrollment  of  approximately 
3000  students  in  grades  9 - 12)  in  the  southwestern  region  of  the  US  were  surveyed  on  their  homework 
practices.  The  school  was  chosen  because  it  is  one  of  the  most  representative  campuses  in  the  state,  with  a 
racial  and  economic  distribution  that  matches  the  state  average:  51%  white,  34%  Hispanic,  7 % African 
American,  4%  Asian,  and  3%  Native  American  and  60%  of  the  students  qualify  for  Free/Reduced  lunch.  This 
study  was  approved  by  the  Institutional  Review  Board  at  the  researchers’  university,  and  permission  to 
collect  the  data  and  report  the  findings  was  granted  by  the  principal  and  the  school  district. 

The  purpose  of  surveying  both  teachers  and  students  was  to  establish  the  nature  of  the  relationship 
between  teacher  policies  and  self-reported  student  help-seeking  activities.  For  instance,  if  teachers  approve  a 
particular  help-seeking  activity,  such  as  taking  and  using  detailed  class  notes,  students  may  be  more  likely  to 
gravitate  toward  using  that  resource.  This  paper  focuses  on  the  results  of  the  student  survey;  the  results  of  the 
teacher  survey  will  be  published  separately. 

The  researchers  asked  all  mathematics  teachers  at  the  school  to  administer  the  student  survey  during  a 
normal  class  session  late  in  the  semester,  and  teachers  did  so  voluntarily.  Classes  at  all  levels  were  surveyed; 
Remedial,  consisting  of  Standard  Mathematics  I and  Skill  Builder;  Core,  consisting  of  Algebra  I,  Algebra  II, 
Geometry,  Trigonometry  and  College  Math,  and;  Advanced,  consisting  of  Honors  Algebra  I,  Honors  Algebra 
II,  Honors  Geometry,  Pre-calculus,  and  AP  calculus.  Findings  are  reported  across  these  different  course  types 
because  we  hypothesized  that  help-seeking  strategies  might  differ  depending  on  how  advanced  the  course 
material  was.  The  anonymous  written  student  survey  consisted  of  four  questions  in  which  students  reported 
on  the  location  where  they  completed  their  homework,  the  type  and  amount  of  help-seeking  resources  that 
they  used,  and  their  perceived  need  for  help.  In  addition,  students  were  asked  about  their  familiarity  with  and 
usage  of  several  popular  digital  resources.  The  survey  was  intentionally  designed  to  be  very  brief  in  order  to 
take  up  only  a minimal  amount  of  instructional  time  and  encourage  participation. 

Exact  binomial  95%  confidence  intervals  were  calculated  for  proportions.  Independence  of  two 
categorical  variables  was  assessed  using  a Chi-squared  test.  Averages  are  calculated  with  linear  regression 
with  bootstrapped  confidence  intervals,  to  accommodate  lack  of  normality  of  residuals.  Significance  is 
reported  at  the  5%  level.  All  data  manipulation  and  statistical  analyses  were  carried  out  in  Stata  MP  vl2 
(StataCorp.,  2012). 


3.  RESULTS 

A total  of  39  teachers  and  2473  students  participated,  giving  a response  rate  of  75%,  with  representation  from 
every  course  with  the  exception  of  Consumer  Mathematics.  Participant  students  were  between  14  and  19 
years  old.  The  overwhelming  majority  of  the  students  (-85%)  were  enrolled  in  core  courses,  only 
approximately  10%  were  taking  advanced  math  courses,  and  very  few  (-4%)  were  enrolled  in  remedial 
courses. 

3.1  Resources 

Figure  1 shows  the  percentage  of  students  across  course  types  (remedial,  core,  and  advanced)  who  reported 
using  a resource  for  seeking  help  and  to  what  degree  the  resource  is  used  for  help  in  a typical  homework 
assignment  (for  two  or  fewer  problems,  for  some  problems,  or  for  most  or  all  problems).  In  Figure  1,  the 
results  are  collapsed  across  frequency  of  use:  The  graph  on  the  left  (never  or  infrequent)  represents  reported 
use  of  a resource  for  two  problems  or  less,  the  graph  in  the  middle  (occasional)  represents  reported  use  for 
some  of  the  problems,  and  the  graph  on  the  right  (frequent)  represents  reported  use  of  a resource  for  most  or 
all  of  the  problems  on  a typical  homework  assignment. 

Below  are  the  major  finding  for  what  resources  students  commonly  use  and  the  extent  to  which  they  use 
them: 
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• Digital  resources  are  much  less  popular  than  traditional,  non-digital  resources  for  students  enrolled 
in  all  levels  of  high  school  mathematics  courses. 

• Students  do  not  generally  seek  help  from  others  at  home  or  from  a tutor. 

• Students  in  core  or  remedial  courses  who  do  have  a tutor  are  more  likely  than  students  in  advanced 
courses  to  ask  for  help  on  substantial  portions  of  their  assignments 
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Figure  1.  Percentage  of  students  who  reported  using  each  resource  on  a typical  homework  assignment,  by  course  level 
(Remedial,  Core  or  Advanced)  and  level  of  usage,  with  95%  exact  binomial  confidence  interval. 

• Students  seek  help  from  friends  and  teachers  more  frequently  than  others  at  home  and  tutors,  and 
this  is  especially  true  for  students  in  remedial  courses. 

• Students  in  core  and  remedial  courses  are  more  likely  than  students  in  advanced  courses  to  seek  help 
from  friends  and  teachers  on  substantial  portions  of  their  assignments. 

• Students  in  advanced  courses  skip  or  copy  solutions  for  only  one  or  two  problems  on  an  assignment, 
whereas  students  in  core  and  remedial  courses  are  more  likely  to  engage  in  these  activities  for 
substantial  portions  of  their  assignments. 

• Textbooks  are  used  infrequently  by  students  at  all  levels,  and  this  is  especially  true  for  students  in 
remedial  courses. 

• Class  notes  are  the  most  popular  resource  for  seeking  help  on  substantial  portions  of  their 
assignments  for  students  enrolled  in  core  and  advanced  courses. 

3.2  Locale 

Table  1 shows  the  percentage  of  students  enrolled  in  difference  levels  of  courses  by  the  locale  in  which  they 
do  their  homework.  Course  level  was  a characterizing  factor  for  where  the  majority  of  homework 
assignments  are  typically  completed. 

Below  are  the  major  findings  for  where  students  complete  their  homework  assignments: 

• Students  in  advanced  courses  generally  complete  their  assignments  at  home  and  very  seldom  during 
class,  whereas  students  in  remedial  courses  complete  large  portions  of  their  assignments  in  class  and 
very  little  of  their  assignments  at  home. 

• Students  in  core  courses  complete  portions  of  their  homework  assignments  in  varied  locations, 
although  the  largest  portions  of  their  assignments  are  completed  at  home. 

• Study  hall  is  not  used  frequently  for  completing  homework  by  students  at  any  level,  and  particularly 
not  by  students  in  remedial  courses. 

• In  general,  students  who  do  homework  in  study  hall  complete  less  than  half  of  their  assignments 
there 
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Table  1 . Percentage  of  students  using  each  location  to  do  a typical  homework  assignment  and  the  amount  of  help  wanted. 
P-values  are  from  a Chi-square  test  of  independence  (Remedial  N=37,  Core  N=1584,  Advanced  N=182). 


Amount  of  homework 

P- 

value 

Location 

Course  Type 

None 

Quarter 

Half 

Most 

Chi-sq 

At  home 

Remedial 

29 

37 

16 

13 

Core 

22 

22 

19 

35 

Advanced 

5.5 

11 

14 

69 

0.00 

In  class 

Remedial 

2.6 

26 

39 

32 

Core 

18 

36 

23 

19 

Advanced 

47 

40 

7.1 

2.7 

0.00 

In  study  hall 

Remedial 

58 

16 

7.9 

11 

Core 

39 

26 

16 

15 

Advanced 

43 

37 

9.9 

6 

0.00 

3.3  Resource  Awareness 

Figure  2 show  the  percentage  of  students  aware  of  various  popular  homework  help  websites.  Below  are  the 
major  findings  for  how  aware  students  are  regarding  several  popular  digital  resources: 

• In  general,  students  are  completely  unaware  of  the  existence  of  several  popular  online  homework 
help  sites. 

• Depending  on  course  level,  some  online  sites  are  more  popular  than  others. 
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Figure  2.  Percentage  of  students  who  reported  having  heard  of  each  online  resource,  hy  course  level  (Remedial,  Core  or 
Advanced),  with  95%  exact  binomial  confidence  interval  (Remedial  N=37,  Core  N=1584,  Advanced  N=182). 

3.4  Need  for  Additional  Help 

Figure  3 depicts  the  relationship  between  the  amount  of  help  needed  and  awareness  of  digital  resources.  In 
general,  students  reported  wanting  more  help  than  they  are  getting.  Below  are  the  major  findings  for  how 
much  additional  help  students  perceive  that  they  need: 
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• In  general,  students  want  more  help  on  their  math  homework  than  they  are  receiving. 

• As  the  level  of  course  decreases  the  perceived  need  for  additional  help  increases. 

• The  amount  of  extra  help  desired  is  independent  of  the  awareness  of  digital  resources  for  students  in 
core  and  advanced  courses. 

• Students  in  remedial  courses  who  are  aware  of  at  least  two  digital  resources  perceive  that  they  need 
less  extra  help  than  those  who  have  only  heard  of  one  at  most  or  none  at  all. 


Figure  3.  Average  amount  of  additional  homework  help  wanted,  by  the  amount  of  online  resources  heard  of,  and  by 
course  level  (Remedial,  Core  or  Advanced),  with  95%  confidence  intervals  estimated  from  bootstrapped  linear  regression 

model  (N=  1,729). 


4.  CONCLUSIONS 
4.1  Limitations 

The  use  of  a brief  survey  instrument  to  investigate  homework  help-seeking  resources  allowed  us  to  gather 
information  from  a large  number  of  students  with  relative  ease.  However,  this  design  choice  has  several 
limitations  that  lead  to  open  questions.  Because  results  were  based  on  self-reporting,  there  is  no  way  to  assess 
the  validity  of  students’  descriptions  of  their  help-seeking  habits.  This  said,  we  are  cautiously  optimistic  that 
students  were  for  the  most  part  honest  in  their  reports;  on  the  questionnaire,  they  freely  admitted  to  the 
unsanctioned  practice  of  copying  problem  solutions  from  others.  In  addition,  though,  even  if  students  were 
straightforward  in  their  responses,  they  may  have  had  differing  interpretations  of  the  terminology  used  in  the 
survey,  such  as  ‘similar  problem  type’  or  ‘online  tutorial.’  Related  to  this,  students  may  only  have  known 
some  of  the  digital  resources  listed  by  their  logo,  and  may  not  have  recognized  them  by  name. 

The  survey  also  did  not  address  questions  about  general  technology  availability  and  use.  The  assumption 
made  was  that  students  would  gravitate  toward  resources  that  were  viewed  as  most  helpful.  However,  it  is 
also  possible  that  circumstances,  beyond  just  the  location  where  homework  is  done,  dictate  how  students  seek 
help.  Items  that  ask  students  to  describe  their  access  to  the  Internet  and  their  use  of  technology  would  be 
informative  additions  to  this  survey. 

Furthermore,  it  would  be  useful  to  consider  how  various  resources  relate  to  achievement.  If  students  are 
able  to  perform  well  in  math  class  by  relying  almost  exclusively  on  their  class  notes  for  homework  help,  then 
they  presumably  have  no  need  for  other  available  resources.  The  results  of  this  study  suggest  that  students 
may  be  learning  through  experience  that  any  ‘important’  information  can  be  found  in  their  notes.  This  calls 
for  an  investigation  of  what  students’  class  notes  look  like,  how  the  content  of  their  notes  compares  with 
class  instruction  as  well  as  with  homework  and  exam  questions,  and  how  all  of  these  factors  relate  to  learning 
and  understanding. 
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4.2  Summary 

This  study  shows  that  students  seem  to  gravitate  toward  using  traditional  rather  than  digital  resources  for 
seeking  homework  help.  At  the  same  time,  although  many  students  express  desire  for  more  help,  they  are 
unaware  of  the  existence  of  several  popular,  freely  available  digital  resources.  These  patterns  of  resource 
familiarity  and  use  may  reflect  students’  beliefs  about  homework,  governed  by  the  level  of  course  they  are 
taking,  as  a school-centered  activity,  as  well  as  the  nature  of  the  resources  themselves.  In  particular,  it  would 
be  worth  researching  what  digital  resources  are  being  used  and  how  by  remedial  courses  since  they  reported 
needing  less  help  as  their  awareness  of  available  resources  increased.  Another  line  of  research  concerns  note- 
taking and  how  this  activity  relates  to  achievement,  particularly  as  students  transition  from  a more  structured 
high  school  setting  to  that  of  university,  and,  even  perhaps  online  instruction. 

In  sum,  these  results  raise  important  issues  about  resources  and  their  use  in  instruction.  It  is  important  for 
instructors  and  researchers  to  discuss  the  role  of  available  resources  for  providing  homework  help,  the  quality 
and  affordances  of  different  resources  - both  digital  and  non-digital,  and  to  find  ways  to  help  support 
students  in  their  use  of  these  resources.  The  point  is  not  whether  students  in  the  ‘Net  generation’  are  doing 
their  homework  differently  than  other  generations  - but  rather  how  we  can  maximize  their  opportunities  to 
develop  deeper  understandings  of  mathematics  as  they  work  on  their  assignments. 
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ABSTRACT 

This  paper  reviews  research  literature  on  the  production  and  commercialization  of  IT-enabled  higher  education  in 
computer  science.  Systematic  literature  review  (SLR)  was  carried  out  in  order  to  find  out  to  what  extent  this  area  has 
been  studied,  more  specifically  how  much  it  has  been  studied  and  to  what  detail.  The  results  of  this  paper  make  a 
contribution  to  research  by  indicating  a gap  in  the  recent  literature  concerning  the  production  and  commercialization  of 
higher  education  on  practical  level. 
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1.  INTRODUCTION 

The  productisation  of  information  technology  (IT)  enabled  higher  education  is  one  way  to  improve  the 
competitiveness  of  educational  organisations,  and  also  the  key  factor  to  the  global  market.  Thus,  IT-enabled 
higher  education  entails  many  opportunities  as  it  requires  careful  planning,  resources  and  adoption  of  the  new 
teaching  culture.  In  order  to  ensure  both  the  quality  and  effectiveness  of  IT-enabled  higher  education, 
educational  institutions  need  to  consider  productization  of  their  higher  education. 

Productization  refers  to  the  modification  of  a material  good  or  an  immaterial  service,  or  their 
combination,  into  a commercial  product  that  can  be  replicated  in  large  quantities  using  a certain  method.  In 
an  educational  context,  the  target  of  productization  may  be  for  example  course  or  program,  training  or 
consulting  service,  learning  materials  and  content,  learning  technologies,  or  related  research  results  and 
services.  Successful  productization  necesitates  that  the  company  identifies  its  core  competence  areas  and  the 
possible  additional  or  supporting  services  (Sipila  2005.) 

A systematic  literature  review  (SLR)  was  carried  out  in  order  to  identify  current  practices  for  the 
productization  of  IT-enabled  higher  education  in  computer  science.  Firstly,  we  present  the  SLR  method 
applied,  and  secondly,  the  results  and,  finally,  conclude  by  discussing  the  resulting  nine  categories  of 
commericalization  and  productization  of  higher  education  found  by  the  SLR. 


2.  RESEARCH  METHOD 

The  research  process  follows  Kitchenham  et  al.  (2010)  by  including  the  following  steps:  defining  the  research 
questions,  describing  the  search  process,  describing  the  how  the  included  papers  were  selected,  quality 
assessment,  and  data  extraction  process.  Other  guidelines  for  conducting  a SLR  (Okoli  and  Schabram  2010) 
were  used  in  a complementary  manner.  The  research  was  carried  out  by  two  researchers  (later  referred  to  as 
researcher  1 and  researcher  2). 
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The  purpose  of  our  study  is  to  identify  studies  reporting  ways  of  productizing  or  commercializing  IT- 
enabled  higher  education  in  computer  science.  Since  we  were  aware  of  only  a few  of  such  studies,  the 
research  question  was  formulated  in  a broad  sense  so  that  it  would  capture  the  aspect  of  how  much  this  area 
has  been  studied  as  well  as  the  aspect  of  how  deeply  it  has  been  studied:  “To  what  extent  the 
commercialization  and  productization  of  higher  education  in  computer  science  and  information  technology 
has  been  studied  so  far?" 

2.1  The  Search  Process 

The  search  process  started  with  word  search  in  electronic  databases.  Researcher  2 searched  eight  digital 
libraries  (on-line  databases)  and  one  automated  search  engine.  The  digital  libraries  included:  ACM  Digital 
Library,  IEEE  Xplore  Digital  Library,  Web  of  Science,  Science  Direct,  Education  Resources  Information 
Center  (ERIC),  EBSCOhost,  InderScience  Publishers,  and  Springer  Link.  Google  scholar  search  engine  was 
used  to  complement  the  above  mentioned  electronic  libraries.  The  search  was  carried  out  between  August 
and  October  2011  and  it  was  limited  to  articles  published  between  2001  and  2011.  The  following  search 
words  were  used:  productization,  commercialization,  higher  education,  information  technology,  computer 
science,  and  information  systems.  The  word  search  was  targeted  at  title,  abstract,  review,  and  keywords.  The 
following  combinations  of  search  words  were  used  as  a base  search  for  each  electronic  library: 

1.  "productization  AND  higher  education  AND  commercialization" 

2.  "commercialization  AND  higher  education  AND  productization" 

3.  "productization  OR  higher  education  OR  commercialization" 

4 . "commercialization  OR  higher  education  OR  productization" 

5.  "productization  AND  higher  education  OR  commercialization" 

6.  "higher  education  AND  productization  OR  commercialization" 

7.  "productization  AND  commercialization  OR  higher  education" 

8.  "commercialization  AND  productization  OR  higher  education" 

9.  "productization  OR  commercialization  AND  higher  education" 

10.  "productization  OR  higher  education  AND  commercialization" 

11.  "commercialization  AND  higher  education  OR  productization" 

12.  "commercialization  OR  higher  education  AND  productization" 

13.  "commercialization  OR  productization  AND  higher  education" 

These  basic  combinations  were  adapted  to  the  particular  search  possibilities  of  each  database.  If  the 
database  did  not  have  automatic  lemmatization  option,  then  the  “z-search”  was  followed  by  identical  search 
with  “s-search”  (e.g.  productisation  instead  of  productization).  Advanced  search  was  used  as  default  if 
provided.  The  details  of  the  literature  search  were  recorded  along  the  whole  search  process  (see  Okoli  and 
Schabram  2010)  in  MS  Excel  file  with  appropriate  metadata. 

2.2  Paper  Selection 

Paper  selection  process  took  place  in  two  phases  (see  Eigure  1).  Eirstly,  researchers  1 and  2 undertook  an 
initial  screening  (Kitchenham  et  al.  2010)  of  the  found  1559  papers.  Papers  were  distributed  to  the 
researchers  by  database  in  a way  that  one  reviewed  all  papers  belonging  to  the  allocated  database.  The 
researchers  read  through  all  the  tittles,  abstracts  and  keywords  individually  and  marked  each  paper  as 
excluded  or  included.  The  researchers  then  discussed  about  the  selected  papers  and  after  reaching  a 
consensus  made  a decision  on  which  articles  to  exclude  and  which  to  include. 

Secondly,  quality  screening  was  performed.  Quality  appraisal  during  the  paper  selection  process  was 
carried  out  as  multiple  coding  by  two  researchers  who  checked  the  suitability  and  relevance  of  the  articles 
regarding  the  topic  of  the  study.  Multiple  coding  was  carried  out  as  parallel  coding  of  the  articles  and 
simultaneous  discussion  about  the  quality  and  content  of  the  articles  (Barbour,  2001).  This  way  both  the 
coding  and  content  of  the  papers  as  well  as  inclusion  and  exclusion  criteria  were  double  checked  during 
paper  selection.  The  abstracts  were  read  thoroughly  and,  if  necessary,  the  whole  paper  was  read  in  order  to 
assess  the  compliance  with  the  inclusion  criteria.  Next,  suggestions  were  made  for  excluding  papers  that 
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appeared  suitable  at  first  glance  but  did  eventually  not  meet  with  the  inclusion  criteria.  Finally,  the  final 
included  papers  were  discussed  and  consensus  was  reached. 


Initial  screening  — > 


Quality  screening  — 


Figure  1.  Paper  selection  process. 

The  criteria  for  inclusion  were  1)  the  paper  has  to  deal  with  higher  education  or  university  education,  2) 
the  issue  of  commercialization  or  productization  had  to  be  present,  and  3)  the  article  has  to  be  published  in  a 
refereed  journal  or  conference  proceedings.  Articles  concerning  the  commercialization  or  patenting  of 
university  research  were  included  for  additional  value.  The  exclusion  criteria  included  that  the  article  1)  was 
concerned  with  lower  level  education  (e.g.,  high  school),  or  2)  was  published  on  a non-refereed  forum. 

The  initial  screening  reduced  the  amount  of  articles  from  1559  to  126.  This  was  followed  by  a 
redundancy  check  after  which  double  articles  were  removed  within  the  results.  This  was  done  in  two  steps; 
firstly  removing  the  doubles  within  each  database  and  then  removing  doubles  across  databases.  After 
removing  double-articles  within  each  database  the  amount  of  articles  was  78.  After  removing  all  the  doubles 
across  the  databases  and  search  engines,  the  amount  of  articles  was  63.  As  a result  of  quality  screening,  23 
articles  were  removed  after  which  the  final  amount  of  included  articles  was  40. 


3.  RESULTS 

In  comparing  the  content  of  the  included  40  papers  to  our  research  question  it  was  obvious  that  the  results  did 
not  directly  match  to  the  focus  of  the  study.  Hence,  a classification  of  the  articles  was  carried  out  in  order  to 
define  their  subject  matter.  The  articles  were  classified  against  the  following  criteria:  distribution  of  articles 
by  publication  forum  (journal,  conference),  and  distribution  of  articles  by  subject  (see  Ngai  and  Gunasekarun 
2007). 

3.1  Distribution  by  Publication  Forum 

The  identified  papers  were  mainly  published  in  refereed  journals  and  conferences.  The  results  include  a total 
of  29  journal  articles  from  21  journals.  The  highest  publishing  results  were  in  Industry  and  higher  education 
(4  articles),  Scientometrics  (4  articles).  Globalisation,  Societies  and  Education  (2  articles)  and  Assistive 
Technology  Outcomes  and  Benefits  (2  articles).  The  rest  of  the  journals  had  published  one  article. 

The  articles  were  distributed  in  journals  from  various  disciplines  (see  Table  1).  Majority  of  the  journal 
articles  (24  articles)  were  published  in  journals  on  higher  education  and  related  fields.  We  classified  these 
journals  in  five  categories,  namely  higher  education,  teaching  and  learning,  education  policy  and  planning, 
educational  technologies,  and  international  education.  The  second  largest  category  of  journals  (7  articles)  was 
based  on  discipline.  It  was  divided  into  subcategories  of  natural  sciences,  relationship  between  science  and 
public,  library  sciences,  and  information  processing  and  analysis.  The  third  category  of  journals  (1  article) 
was  concerned  with  European  policies. 
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Table  1.  Distribution  of  papers  in  journals. 


Subject  Category 

Journal  name 

Number  of  articles 

1.  Education 

(24) 

1 . 1 Higher  Education 

Industry  and  higher  education 

4 

Research  in  higher  education 

1 

Higher  Education  Quarterly 

1 

Thought  & Action 

1 

Journal  of  further  and  higher  education 

1 

Higher  education 

1 

1.2  Teaching  and  learning 

Journal  of  University  Teaching  and  Learning  Practice 

1 

e-Journal  of  Business  Education  & Scholarship  of 
Teaching 

1 

International  Journal  of  Learning  and  Change 

1 

Learning  and  Teaching:  The  International  Journal  of 
Higher  Education  in  the  Social  Sciences 

1 

1.3  Education  policy  and  planning 

Higher  Education  Policy 

1 

Journal  for  critical  education  policy  studies 

1 

The  International  Journal  of  Higher  Education  and 
Educational  Planning 

1 

Journal  of  Higher  Education  Policy  and  Management 

1 

1 .4  Educational  technologies 

Assistive  Technology  Outcomes  and  Benefits 

2 

1 .5  International  education 

Globalization,  Societies  and  Education 

2 

Russian  education  and  society 

1 

Journal  of  Studies  in  International  education 

1 

2.  Information  sciences 

(5J 

2.1  Library  sciences 

Scientometrics 

4 

2.2  Information  processing  and  analysis 

Scientific  and  Technical  Information  Processing 

1 

3.  European  policies 

(1) 

3.1  Planning  studies 

European  planning  studies 

1 

Total 

29 

The  results  included  11  conference  papers  (Table  2).  There  was  a great  variety  in  the  field  and 
distribution  of  the  conferences,  which  were  categorized  according  to  their  subject  area  in  the  following 
categories:  computer  science,  engineering,  information  technology,  technology  management,  education 
management,  and  business  management. 

Table  2.  Distribution  of  papers  in  conferences. 


Subject  category  Conference  name  Number  of  articles 

Computer  science  IEEE  International  Conference  on  Fuzzy  Systems  1 

Engineering  Canadian  Conference  on  Electrical  and  Computer  Engineering  1 

IEEE  International  Conference  on  System  Engineering  and  Technology  1 

IEEE  International  Conference  on  Industrial  Engineering  and  Engineering  1 

Management 

Information  technology  ICO  International  Conference  on  Information  Photonics  1 

International  Conference  on  Communication  Systems,  Networks  and  1 

Applications 

International  Convention  on  Information  and  Communication  Technology,  1 

Electronics  and  Microelectronics 

Technology  management  IEEE  International  Technology  Management  Conference  1 

International  Conference  on  Education  Technology  and  Computer  1 

Education  management  International  Conference  on  Education  and  Management  Technology  1 

Business  management  International  Conference  on  Business  Management  and  Electronic  Information  1 

Total  11 


The  distribution  of  articles  by  publication  forum  indicates  that  our  topic  was  over  three  times  more 
popular  in  Journals.  The  high  dispersion  in  both  journals  and  conferences  indicates  that  there  is  no  established 
journal  for  the  topic. 
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3.2  Distribution  by  Subject 

The  articles  were  classified  according  to  their  subject  content.  Nine  different  classes  were  recognized: 
commercialization  of  research,  technology  transfer,  commercialization  of  education,  educational  reform, 
knowledge  transfer,  patenting,  educational  system,  academic  entrepreneurship,  and  technology 
commercialization  effects  (Table  4). 


Table  4.  Classification  of  papers  by  subject. 


Class  # 

Class  name 

Number  of 
papers 

References 

1 

Commercialization  of 
research 

8 

Dube  and  Lisk  (201 1),  Harman  (2010),  Hussain  et  al. 
(2010),  Kerr  (2011),  Li  and  Xia  (2011),  Smith  (2010), 
Suvinen  et  al.  (2010),  Yang  and  Chang  (2010) 

2 

Technology  transfer 

7 

Fraser  (2010),  Gonnova  and  Kavtaradze  (2010),  Lane 
(2010),  Leahy  and  Lane  (2010),  Prokopiev  (2011), 
Somsuk  (2010),  Wang  and  Cao  (2010) 

3 

Commercialization  of 
education 

6 

Arnold  (2010),  Naidoo  (2010),  Robertson  (2010),  Tail 
(2010),  Tsai  and  Yang  (2010),  Wong  and  Westwood 
(2010) 

4 

Educational  reform 

5 

Aboites  (2010),  Graeme  and  Peim  (2011),  Liu  and 
Crossley  (2010),  Raciti  (2010),  Susanti  (2011) 

5 

Patenting 

4 

Benneworth  and  Jongbloed  (2010),  Klitkou  and 
Gulbrandsen  (2010),  Leydesdorf  and  Meyer  2010), 
Wong  and  Singh  (2010) 

6 

Educational  system 

4 

Allen  (2010),  Dubin  and  Zorkaia  (2010),  Shore  (2010), 
Yassin  et  al.  (2011) 

7 

Knowledge  transfer 

2 

Bezic  et  al.  (2011),  Kuiken  and  van  der  Sijde  (2011) 

8 

Academic 

entrepreneurship 

2 

Arapostathis  (2010),  Goldstein  (2010) 

9 

Technology 

commercialization  effects 

2 

Powers  and  Campbell  (2010),  Wong  and  Singh  (201 1) 

Commercialization  of  research  class  includes  nine  papers  on  the  enhancement  or  measurement  of 
commercialization  of  academic  research.  It  includes,  for  instance,  framework  through  which  the  researchers 
can  engage  commercialization  into  the  research  process  as  a vital  part  of  it  (Smith  2010),  perceptions  of  the 
success  of  innovation  and  research  commercialization  in  Australia  (Harman  2010),  and  the  role  of 
intermediary  organizations  (Suvinen  et  al.  2010)  or  university  lawyers  (Hussain  et  al.  2010)  in  the  process  of 
commercialization  of  academic  research.  Further,  it  includes  studies  on  the  influence  of  entrepreneurial 
commitment  (Yang  and  Chang  2010)  and  federal  funding  (Kerr  2011)  or  governmental  programs  (Dube  and 
Lisk  2011)  on  research  commercialization.  Also,  the  aspects  of  knowledge  commercialization  modes  and 
effects  in  universities  (Li  and  Xia  2011)  were  included.  The  papers  in  this  class  focus  on  the  means  of 
enhancing  or  measuring  the  commercialization  of  university  research  in  general  and  do  not  provide  results  on 
the  specific  area  of  commercialization  of  higher  education. 

Technology  transfer  class  includes  seven  papers  that  explain  how  to  enhance,  understand  and  analyse  the 
academic  technology  transfer.  These  include  factors  explain  factors  influencing  improving  university 
technology  transfer  (Wang  and  Cao  2010),  analysis  of  university-industry  technology  transfer  programme's 
success  (Somsuk  2010),  measurement  and  enhancement  of  the  impacts  of  technology  transfer  (Fraser  2010), 
enhancement  of  technology  transfer  and  commercialization  by  business  games  (Gonnova  and  Kavtaradze 
2010),  description  of  how  Canadian  innovation  and  commercialization  program  enhances  industry-academia 
technology  transfer  (Prokopiev  2011),  the  barriers  and  carriers  of  successful  technology  transfer  of  assistive 
technology  devices  in  USA  (Leahy  and  Lane  2010),  and  the  confluence  of  academic  research  of  technology 
transfer  of  assistive  technologies  for  value  creation  (Lane  2010).  These  papers  focus  on  technology  transfer 
and  not  on  education. 

Commercialization  of  education  class  includes  six  papers  on  the  commercialization  of  higher  education  in 
the  changing  global  business  context.  Naidoo  (2010)  examines  the  export  readiness  of  universities  and  its 
implications  on  export  recruitment  of  students.  Robertson  (2010)  studies  the  commercialization. 
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corporatization  and  competitiveness  of  U.K.  universities  emerging  from  the  transformation  of  higher 
education  toward  globally  competitive  education  system.  Wong  and  Westwood  (2010)  provide  empirical 
findings  from  the  conflicting  interests  and  tension  that  the  commercialization  pressure  have  brought  at 
universities.  Arnold  (2010)  addresses  the  same  issue  from  university  teacher’s  perspective  while  Tsai  and 
Yang  (2010)  discuss  about  the  role  and  profession  of  teachers  in  commercialized  education.  Tait  (2010) 
studies  the  commercialization  of  education  from  the  perspective  of  Chinese  student  in  Western  university 
format.  These  papers  provide  insights  on  the  impact  of  the  global  commercialization  pressure  on  university 
education.  They  fail  to  provide,  however,  views  how  to  productize  or  commercialize  higher  education. 

Educational  reform  class  includes  five  papers  discussing  the  strategic  and  political  aspects  of  educational 
system  reforms  and  related  commercialization  of  higher  education  including  experiences  on  marketing  the 
university  programmes  in  China  (Liu  and  Crossley  2010),  analysis  of  the  privatization  and  marketization  of 
higher  education  in  Indonesia  (Susanti  2010),  the  implication  of  the  reform  and  commercialization  of 
Australian  higher  education  (Raciti  2010),  a critical  analysis  of  the  negative  implications  of  Bologna  Process 
on  Latin  American  universities  (Aboites  2010),  and  results  of  the  quality  of  exported  higher  education 
(Graeme  and  Peim  2011).  These  papers  provide  knowledge  of  national  university  reforms  that  have  shifted 
higher  education  from  traditional  research  to  commercial  business. 

Patenting  class  includes  four  articles  concerning  the  relationship  between  academic  patenting  and 
scientific  publishing  (Klitkou  and  Gulbrandsen  2010,  Wong  and  Singh  2010),  the  development  of  university 
patenting  due  to  external  forces  such  as  Bayh-Dole  (Leydesdorf  and  Meyer  2010)  and  the  unequal  position  of 
soft  sciences  in  comparison  to  hard  sciences  in  relation  to  patenting  and  commercialization  (Benneworth  and 
Jongbloed  2010).  In  this  class,  commercialization  is  related  activities  leading  to  patenting  while  academia  is 
defined  as  a source  for  material  for  patenting. 

Educational  system  class  includes  four  articles  reporting  the  current  state  of  national  educational  systems. 
Survey  results  of  Dubin  and  Zorkaia  (2010)  show  that  in  Russia  the  higher  education  has  social  value.  Yassin 
et  al.  (2011)  presents  an  instrument  for  the  evaluation  of  research  performance  and  results  of  its  use  in 
Malaysian  universities.  Shore  (2010)  analyses  the  challenges  of  the  modern  university  due  to  contradictory 
requirements  between  the  traditional  and  new,  economic  and  strategic  pressures.  Allen  (2010)  analyses  the 
Royal  Society’s  suggestions  for  enhancing  science  in  UK  and  contrasts  them  to  United  States.  The  papers  in 
this  class  address  the  issues  of  commercialization  and  higher  education  in  national  context  by  describing  the 
country’s  educational  systems  with  its  characteristics  and  recent  developments. 

Knowledge  transfer  includes  two  articles  on  the  transition  of  scientific  knowledge  from  academia  to 
public  sector  and  commercialization  of  such  knowledge.  Kuiken  and  van  der  Sijde  (2011)  study  the 
commercialization  and  dissemination  of  academic  knowledge  by  using  communication  theory  in  interpreting 
the  results.  Their  research  provides  topics  for  further  research  in  relation  to  dissemination  capacity.  Bezic  et 
al.  (2011)  discuss  about  the  role  of  Croatian  national  technology  transfer  office  as  a channel  for  transferring 
university  research  results  into  economy.  These  studies  include  some  elements  of  commercialization  and 
higher  education  without  focusing  on  commercialization  process  of  higher  education. 

Academic  entrepreneurship  class  consists  of  two  papers  studying  the  role  of  entrepreneurship  within 
research  universities  and  national  politics.  They  focus  on  the  position  of  academic  entrepreneurship  with 
respect  to  national  politics  and  innovation  policies  (Arapostathis  2010),  and  on  national  attitudes  toward 
entrepreneurship  and  commercialization  of  university  (Goldstein  2010).  The  aspect  of  commercialization  in 
these  articles  focuses  on  university  research  and  innovation  in  relation  to  entrepreneurship. 

Technology  commercialization  effects  class  includes  two  studies.  Powers  and  Campbell  (2010)  argue  that 
in  U.S.  universities  the  licencing  or  patenting  of  technologies  do  not  have  an  effect  on  research  productivity 
research.  Wong  and  Singh  (2011)  examine  university-industry  collaboration  by  measuring  the  amount  of 
university-industry  co-publications  on  technology  commercialization.  These  papers  do  not  take  a stand  on  the 
ways  or  extent  to  which  IT-enabled  higher  education  could  be  commercialized  or  productized. 


4.  CONCLUSIONS 

The  productization  of  IT-enabled  higher  education  is  one  solution  for  improving  the  competitiveness  of 
educational  organisations,  and  to  respond  to  the  increasing  demand  for  globalizing  the  higher  education.  We 
reviewed  the  current  literature  in  order  to  identify  ways  in  which  the  productization  of  IT-enabled  higher 
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education  in  computer  science  has  been  done.  The  SLR  yielded  40  papers,  including  29  journal  articles  and 
11  conference  papers.  The  articles  were  classified  according  to  their  study  subject  into  nine  categories: 
academic  entrepreneurship,  patenting,  educational  system,  knowledge  transfer,  educational  reform, 
commercialization  of  research,  commercialization  of  education,  technology  transfer,  and  technology  transfer 
effects.  Unfortunately,  the  results  did  not  provide  the  answer  that  we  were  looking  for.  The  results  show  that 
commercialization  and  productization  of  higher  education  has  been  studied  on  institutional,  national  and 
international  level.  However,  studies  concerning  productization  of  courses,  learning  technologies,  contents  or 
learning  materials,  were  not  found.  Based  on  this,  we  claim  that  there  is  a gap  in  research  that  requires  further 
studying. 
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ABSTRACT 

The  purpose  of  this  paper  is  to  explore  the  effectiveness  of  Action  Design  Research  (ADR),  a combination  of  Action 
Research  and  Design  Science  Research,  as  a methodology  to  examine  how  the  implementation  of  e-leaming  will  affect 
the  learning  outcomes  for  staff  training  in  an  organisational  context.  The  research  involves  an  intervention  in  the  finance 
course  offerings  for  staff  training  in  the  workplace.  The  paper  briefly  introduces  the  research  problem  of  an  inflexible  and 
inefficient  finance  training  offering,  and  the  proposed  solution  of  implementing  e-leaming.  It  then  describes  the  seven 
principles  of  ADR  in  detail,  in  a tabular  format,  with  particular  reference  to  their  application  in  this  study.  The  paper  then 
summarises  how  the  seven  principles  fit  into  the  four  stages  of  ADR,  namely,  (1)  problem  formulation,  (2)  building, 
intervention  and  evaluation,  (3)  reflection  and  learning,  and  (4)  Formalisation  of  learning.  After  discussing  the  ADR 
principles  and  how  they  relate  to  the  research  study,  the  paper  concludes  with  the  suitability  and  advantages  of  adopting 
an  ADR  approach  to  e-learning  research.  These  benefits  appear  to  be  meeting  the  challenge  of  IS  as  an  applied  discipline, 
by  implementing  a solution  in  a real  world  situation,  whilst  also  adding  to  e-leaming  theory  and  academic  knowledge. 


KEYWORDS 

Action  Design  research.  E-learning,  Organisational  training 


1.  INTRODUCTION 

In  the  light  of  the  need  to  accommodate  the  continually  changing  business  environment,  the  swift 
dissemination  of  relevant,  up-to-date  information  is  essential.  When  using  an  integrated  financial  system, 
changes  in  the  system  necessitate  almost  instantaneous  communication,  and  are  often  accompanied  by 
training  issues.  Apart  from  system  changes,  there  is  a basic  requirement  that  the  workforce  have  a working 
knowledge  of  the  system,  and  the  skill  to  extract  relevant  information  from  it.  If  as  a consequence  of 
insufficient  or  ineffective  staff  training,  employees  are  not  skilled  or  technically  capable,  then  a major  source 
of  competitive  advantage  is  lost,  and  the  organisation’s  workforce  cannot  realise  its  full  potential  as  a source 
of  intellectual  capital.  Additionally,  there  appears  to  be  a need  for  more  input  from  IS  research  into  the 
practice  of  e-learning  in  the  workplace,  specifically  its  effectiveness  and  on-going  usage. 

The  immediate  objective  of  this  research  is  to  convert  the  current  instructor-led  finance  training  courses 
for  employees  at  an  organisation  to  computer-based  courses.  The  long-term  objective  is  to  create  a learning 
environment  that  meets  both  the  business  need  for  a knowledgeable,  skilled  body  of  staff,  as  well  as  the 
individual  adult  learner’s  need  for  meaningful,  practical  and  flexible  instruction  (Figure  1).  The  current 
instructor-led  training  is  rigid  as  the  venue  and  the  trainer  need  to  be  booked,  and  the  training  schedule  is  set 
up  a year  in  advance.  This  causes  problems  for  trainees,  trainers  and  line-managers.  Having  established  that 
there  is  a problem  situation,  the  difficulty  is  to  employ  a research  methodology  that  has  a practical  outcome 
for  the  organisation,  whilst  simultaneously  meeting  academic  standards  of  credibility,  dependability  and 
transferability. 
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Training  Intervention  Research  Summary 

• WHO?  Specific  members  of  the  workforce  that  need  access  to  financial  system 

• HOW?  Computer  based  training  courses  (e-learning) 

• WHY?  Learn  and/or  upgrade  technical  skills  & financial  knowledge  required  to 
operate  the  financial  system,  as  well  as  interpret  financial  reports  (Business  need) 
Meaningful,  practical  & flexible  instruction  (individual  adult  learner's  need) 

• LONG  TERM  WHY?  Increased  competitive  advantage  for  organisation 

Figure  1 . Proposed  research  to  investigate  the  change  from  classroom  based  to  computer-based  training  for  compulsory 

finance  training 

There  has  been  debate  as  to  the  nature  of  Information  Systems  (IS)  research,  and  where  it  situates  itself, 
for  some  time  (Lee,  2010;  March  & Niederman,  2012;  Rosemann  & Vessey,  2008;  Walsham,  2012).  Lee 
(2010)  examined  key  concepts  of  IS  research  and  concluded  that  the  future  of  IS  research  may  lie  in  the 
“sciences  of  the  artificial”,  as  distinguished  from  the  “sciences  of  the  natural”  by  Simon  (1969).  The  over- 
riding knowledge  requisite  for  this  type  of  discipline  is  “its  efficiency  and  effectiveness  for  bringing  into 
existence  an  artefact  needed  to  solve  a given  problem,  achieve  a given  goal,  or  otherwise  fulfil  a given  need 
that  is  facing  people  in  the  real  world”  (Lee,  2010,  p346).  This  is  an  opposite  view  to  that  of  the  sciences  of 
the  natural,  which  seek  to  theorise  about  things  that  are  already  there.  As  the  objective  of  this  research  is  to 
intervene  in  a real  world  setting,  and  to  make  changes  based  on  a designed  artefact,  two  research  approaches 
were  considered,  namely  Design  Research  (DR),  also  known  as  Design  Science  Research,  and  Action 
Research  (AR). 

Arising  from  the  debate  regarding  the  IS  discipline  and  how  it  should  differentiate  itself  from  other 
disciplines,  such  as  Computer  Science,  it  may  be  suggested  that  the  IS  discipline  stands  at  the  intersection  of 
people,  organisations  and  technology  (Hevner  et  ak,  2004;  Lee,  2010;  March  & Niederman,  2012). 
According  to  livari  (2007),  Design  Research  (DR)  is  problem  focused  and  seeks  to  design  an  innovative 
product,  or  artefact,  that  addresses  unsolved  problems  within  an  organisation.  In  the  context  of  this  paper, 
artefacts  are  defined  as  software  and/or  hardware  that  are  assembled  to  form  the  object  to  be  implemented. 
This  “object”  inscribes  certain  organisational  structures  into  its  form  as  it  emerges,  involving  more  than  just 
technology  in  its  conceptualisation  and  goals.  Sein  et  al.  (2011,  p.38)  describe  artefacts  as  “ensembles 
emerging  from  design,  use,  and  ongoing  refinement  in  context”.  There  is  a build  and  evaluate  process  which 
forms  a loop  which  is  usually  iterated  as  the  design  is  refined,  before  the  final  artefact  is  produced  (Hevner  et 
ak,  2004).  DR  must  also  contribute  to  research  by  making  “clear  and  verifiable  contributions  in  the  areas  of 
the  design  artefact,  design  foundations  and/or  design  methodologies”  (Hevner  et  ak,  2004,  p83).  However,  a 
possible  limitation  of  design  science  research  is  that  the  building  of  the  artefact  is  separate  from  the 
evaluation  step,  and  the  value  of  the  research  lies  in  its  ability  to  solve  the  original  problem,  rather  than 
testing  it  in  a real  life  setting  (Baskerville  et  ak,  2009;  Sein  et  ak,  2011).  The  intervention  or  introduction  of 
the  artefact  into  the  organisation  is  a secondary  factor  for  much  design  science  research,  as  the  primary  factor 
concerns  the  actual  design  of  the  artefact  and  testing  is  often  not  taken  out  of  the  laboratory  (Cole  et  ak, 
2005). 

According  to  Baskerville  & Wood-Harper  (1996,  p.243),  “Action  research  is  regarded  by  many  as  the 
ideal  post-positivist  social  scientific  research  method  for  IS  research”.  They  base  this  statement  on  the 
premise  that  IS,  being  a highly  applied  field,  requires  integration  with  the  psychological  and  social  facets  of 
changing  or  introducing  a technology  into  the  workplace.  People  are  directly  affected  by  these  changes,  and 
both  monitoring  and  evaluating  their  reactions  should  perhaps  be  considered  an  important  part  of  the 
research.  It  seems  that  acting  on  feedback  or  evaluations  from  end-users  is  an  important  part  of  a successful 
implementation,  and  action  research  would  appear  to  be  ideally  suited  to  allow  for  this.  In  order  to  increase 
user  acceptance  of  the  online  courses,  it  is  felt  that  feeding  user  evaluations  and  comments  directly  back  into 
the  course  design  would  be  advantageous  for  both  the  trainees  and  the  organisation.  Action  research  involves 
a strategy  of  formulating  theory  (Baskerville  & Pries-Heje,  1999)  and  the  researcher  not  only  intervenes  but 
participates  in  the  study,  which  supports  an  interpretive,  anti-positivist  philosophy.  Action  research  is 
iterative  in  nature,  and  the  implemented  process  or  object  is  refined  during  the  cycles.  It  also  includes  what 
McKay  and  Marshall  (2001)  refer  to  as  the  duality  of  action  research,  i.e.  the  researcher  is  both  participant 
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and  observer.  The  researcher  is  required  to  keep  both  roles  in  mind,  and  separated,  as  the  study  progresses. 
On  the  one  hand,  the  researcher  is  acting  as  the  manager  of  the  intervention,  and  as  such  is  directly  involved 
in  the  intervention,  whilst  at  the  same  time,  the  researcher  is  required  to  reflect  on  the  process  in  order  to 
answer  the  research  questions  and  formulate  theories  or  themes.  According  to  McNiff  and  Whitehead  (2006, 
pi 3),  “The  purpose  of  action  research  is  to  generate  living  theories  about  how  learning  has  improved  practice 
and  is  informing  new  practices”.  However,  action  research  has  frequently  been  criticised  for  the  occurrence 
of  personal  bias  or  over-involvement  of  the  researcher,  lack  of  rigour,  and  has  sometimes  been  labelled  as 
consulting  rather  than  researching  (Baskerville  & Wood-Harper,  1996). 

The  purpose  of  this  paper  is  to  attempt  to  address  the  reported  limitations  of  DR  and  AR  research 
approaches,  by  adopting  the  research  method  proposed  by  Sein  et  al.  (2011).  It  is  hoped  that  Action  Design 
Research  (ADR)  will  be  effective  in  addressing  an  actual  problematic  situation  in  an  organisational  setting. 
The  research  encompasses  the  building  of  an  innovative  Information  Technology  (IT)  artefact,  whilst 
allowing  feedback  from  the  users  and  the  organisation  to  influence  the  construction  thereof  in  reiterative 
cycles,  allowing  both  learning  from  the  intervention,  and  simultaneously  producing  academic  theory. 

This  paper  may  be  of  interest  to  any  researchers  seeking  a research  approach  that  meets  the  need  for 
relevance  and  immediacy  in  an  organisational  setting,  whilst  also  contributing  to  empirical  research  on  the 
effectiveness  or  otherwise  of  e-learning,  and  adding  to  the  body  of  knowledge  regarding  theories  of 
organisational  learning  and  training.  The  need  for  this  type  of  IS  research  is  endorsed  by  Baskerville  and 
Wood-Harper  (1996)  from  the  AR  perspective,  and  Hevner  et  al.  (2004)  from  that  of  the  DR  perspective,  and 
Cole  et  al.  (2005),  who  investigated  the  use  of  both  research  methodologies. 


2.  ACTION  DESIGN  RESEARCH 

Cole  et  al.  (2005)  feel  it  would  be  proactive  of  IS  researchers  to  consider  using  both  design  science  research 
and  action  research  together  in  order  to  achieve  a rigorously  designed  artefact  that  is  evaluated  in  a real  life 
organisational  context  to  solve  or  to  ameliorate  a perceived  problem  within  that  organisation.  ADR,  as 
described  by  Sein  et  al.  (2011,  p.40),  “is  a research  method  for  generating  prescriptive  design  knowledge 
through  building  and  evaluating  ensemble  IT  artefacts  in  an  organizational  setting”.  It  has  been  designed  to 
address  the  challenge  of  assisting  IS  practitioners  by  intervening  in  real  world  situations,  whilst  also  building 
theory  that  is  academically  rigorous.  This  also  serves  to  answer  the  call  of  making  IS  research  relevant  to 
practitioners  and  other  IS  professionals,  and  “promoting  engaged  scholarship  through  action  and  design” 
(Conboy  et  al.,  2012,  pi  14).  It  seeks  to  overcome  the  perceived  limitations  of  design  science  research  (DR) 
and  action  research  (AR),  as  well  as  addressing  livari’s  (2007)  concerns  regarding  the  differences  between 
the  two  methods.  By  placing  the  IT  artefact  at  the  centre,  this  methodology  also  addresses  Orlikowski  and 
lacono’s  (2001,  p.l30)  call  to  “increase  attention  and  explicit  consideration  of  IT  artifacts  in  all  (IS)  studies”. 
ADR  has  strict,  explicit  principles  which  are  sometimes  lacking  in  AR,  whilst  the  iterations  and  simultaneous 
building,  intervention  and  evaluation  address  the  sequencing  difficulties  of  DR,  when  attempting  to  use  the 
designed  artefacts  in  organisational  settings.  By  not  taking  the  setting  or  context  of  the  organisation  into 
account,  whilst  designing  the  artefact,  DR  does  not  usually  capture  its  emergent  nature.  There  is 
interdependence  between  design  and  use  in  the  organisation,  which  needs  to  be  captured  or  inscribed  into  the 
artefact.  This  interdependence  is  highlighted  by  ADR. 

There  are  four  stages  in  the  ADR  method.  These  are  (1)  problem  formulation;  (2)  building,  intervention 
and  evaluation;  (3)  reflection  and  learning,  and  finally,  (4)  formalisation  of  learning.  It  is  important  to  note 
that  the  four  stages  are  not  engaged  within  a step  or  waterfall  sequence,  but  are  cyclical  and  reiterative.  Sein 
et  al.  (2011)  also  describe  seven  principles  of  ADR.  These  principles  are  detailed  in  Table  1 below,  with 
particular  application  to  this  research. 
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Table  1.  The  seven  principles  of  ADR,  as  applied  to  this  research 


Principle 

Description  (Sein  et  al.,  2011) 

Application 

1 

Practice- 

inspired 

research 

The  problem  is  drawn  from  the 
real-world  and  is  used  to  create 
knowledge  regarding  a class  of 
problems,  which  are  typifled  by 
this  particular  problem 

Real  world  problem  Need  to  accommodate  continually  changing  business 
environment  by  swift  dissemination  of  relevant,  up-to-date  information. 

Class  of  problem  - effective,  efficient  use  of  e-leaming  as  a tool  that  is 
accepted  by  both  the  business  and  the  trainees. 

This  problem  - converting  current  instructor- led  training  to  online  training  that 
continues  to  meet  business  need  for  knowledgeable,  skilled  workforce,  whilst 
accommodating  trainee’s  need  for  practical  and  flexible  instmction. 

2 

Theory- 

ingrained 

artefact 

The  original  desigji  of  the  artefact 
is  based  on  existing  theories  and 
technologies,  but  the  design  may 
change  once  it  is  evaluated  within 
the  organisation. 

Extensive  literature  review  of  current  e-leaming  body  of  knowledge  resulting  in 
an  artefact  (the  online  training  module)  that  enconpasses  many  success  fectors 
identified,  and  allows  for  learner  feedback  and  evaluation.  Review  covers 
system  design,  system  delivery  and  system  outcome;  learning  approaches  and 
strategies. 

3 

Reciprocal 

shaping 

This  is  where  the  IT  artefact  and 
the  organisational  domain  influence 
each  other,  causing  an  iterative 
cycle  to  be  set  in  motion.  The 
design  of  the  artefact  will  change 
as  it  is  used  in  the  organisational 
setting,  and  the  organisation  may 
be  changed  by  the  use  of  the 
artefrct. 

A^ha  version  of  artefact  tested  by  other  trainers,  changes  to  module  made 
based  on  feedback.  Beta  version  launched  in  controlled  conditions  (le.  in  lab, 
with  instmctor  present,  but  all  other  conditions  of  online  environment  met  - 
duration,  test  taken  at  learner’s  discretion).  Further  changes  made  to  module 
based  on  learner  feedback.  Appropriately  trained,  motivated  and  enabled 
workforce  has  positive  effect  on  organisation’s  financial  reporting,  leading  to 
more  modules,  with  targeted  objectives,  being  offered  online. 

4 

Mutually 

influential 

roles 

This  refers  to  how  the  multiple 
participants  in  the  ADR  project 
share  their  specialised  knowledge 
and  learn  from  each  other.  The 
researchers  bring  in  the  theory  and 
the  practitioners  bring  in  the  work 
practices. 

Researcher  designs  course  module  using  knowledge  of  best  practices  gained 
from  literature  on  e-leaming,  with  input  from  practitioners  regarding  course 
content,  and  specific  working  environment.  With  reiterative  cycles,  workforce 
input  also  influences  shaping  of  module. 

5 

Authentic 

and 

concurrent 

evaluation 

This  means  that  the  process  of 
evaluation  is  never  separated  from 
the  building  and  intervention,  but  is 
an  integral  part  of  both.  Depending 
on  the  form  of  the  artefact, 
evaluation  can  be  either  formative 
or  summative,  but  must  always  be 
allowed  to  occur  spontaneously  in 
the  organisational  context,  rather 
than  in  a controlled  setting. 

The  initial  course  module  is  built  and  offered  to  trainees,  in  working 
environment.  Feedback  and  comments  are  discussed  by  researcher  and 
trainers,  and  changes  made  to  artefact  as  appropriate.  Unexpected 
consequences  are  immediately  visible,  and  can  be  acted  upon  promptly.  The 
artefect  is  continuously  being  evaluated  and  inplemented  in  on-going  wave  in 
dynamic  environment. 

6 

Guided 

emergence 

This  highlights  the  interaction 
between  the  initial  design  and  how 
its  implementation  in  the 
organisational  setting  causes  a 
continuous  evolvement  and  re- 
working of  the  emerging  artefect. 

The  artefact  Is  rolled  out  into  the  real  life  situation,  and  feedback  from 
participants  and  assisting  practitioners  is  immediately  acted  on  by  course 
designer/researcher,  with  re-designed  module  being  offered  almost 
immediately.  Feedback  will  arise  from  course  content  and  organisational 
environment,  le.  equipment,  setting,  timing,  control  Each  cycle  will  result  in  a 
re-working  of  the  artefact  (course  module)  as  required  by  the  evaluations. 

7 

Generalized 

outcomes 

This  ensures  that  the  learning  that 
has  occurred  during  the  iterations 
is  developed  so  that  the  solution  of 
the  specific  problem  can  be 
generalised  to  a class  of  problems. 

The  researcher  will  describe  the  learning  and  re-desigji  of  arteiact  based  on 
user  feedback  and  practitioner  evaluation.  This  learning  will  in  turn  be  linked 
back  to  current  literature  on  best  practices  of  e-leaming  in  organisational 
environment. 
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The  seven  principles,  detailed  in  Table  1,  are  aligned  with  the  ADR  method  stages,  and  should  be  realised 
during  the  relevant  stage.  Thus,  the  first  stage  of  problem  formulation  is  addressed  by  the  principles  of 
“practice-inspired  research”  and  “theory-ingrained  artefact”,  the  second  stage  of  building,  intervention  and 
evaluation  is  addressed  by  the  principles  of  “reciprocal  shaping”,  “mutually  influential  roles”  and  “authentic 
and  concurrent  evaluation”.  The  third  stage  of  reflection  and  learning  is  addressed  by  the  principle  of  “guided 
emergence”,  and  the  fourth  stage,  formalization  of  learning,  is  addressed  by  the  principle  of  “generalized 
outcomes”  (Figure  2,  Sein  et  al.,  201 1). 


3.  Reflection 

and  Learning 


Principle  6.  Guided 
Emergence 


4.  Formalization  of 
Learning 


Pnrcpte  7:  Generalized  Outcomes 


Figure  2.  The  ADR  method:  Stages  and  principles  (Sein  et  al.,  2011) 

[Figure  2 from  M.  K.  Sein,  O.  Henfridsson,  S.  Purao,  M.  Rossi,  and  R.  Lindgren,  “Action  Design  Research,”  MIS 
Quarterly  (35:1),  2011,  p.  41.  Copyright  © 2011,  Regents  of  the  University  of  Minnesota.  Reprinted  by  permission] 

The  perceived  problem  which  is  to  be  investigated  can  arise  from  many  sources  within  the  organisation, 
such  as  practitioners,  end-users,  and  the  existing  technology.  In  this  research,  the  problems  with  the  current 
classroom  based  training  have  been  remarked  on  by  trainees,  trainers,  line-managers  and  financial  managers. 
To  summarise  the  four  stages,  with  the  inherent  principle(s),  the  first  stage  of  ADR,  “problem  formulation”, 
requires  that  the  researcher  identifies  and  conceptualises  the  research  opportunity;  formulates  initial  research 
questions;  casts  the  problem  as  an  instance  of  a class  of  problems;  identifies  contributing  theoretical  bases 
and  prior  technological  advances;  secures  long-term  organisational  commitment;  and  sets  up  roles  and 
responsibilities  (Sein  et  al.,  2011,  p.41).  The  aim  of  the  principle  of  practice-inspired  research  is  that  the 
researcher  produces  “knowledge  that  can  be  applied  to  the  class  of  problems  that  the  specific  problem 
exemplifies”  (p.40).  The  principle  of  a theory-ingrained  artefact,  which  is  also  aligned  to  this  stage,  ensures 
that  the  artefact  to  be  designed  and  tested  within  the  organisation  is  based  on  current  theory,  or  theories. 

These  theories  should  assist  with  the  problem  formulation,  the  identification  of  possible  solutions  and  the 
initially  created  artefact,  which  is  then  introduced  into  the  organisation  for  evaluation,  as  part  of  the  second 
stage,  “building,  intervention  and  evaluation”  (BIE).  This  introduction  of  the  artefact  into  the  real  world 
situation  should  result  in  further  cycles  of  re-designing  or  re-shaping,  and  re-evaluation  of  the  artefact.  Where 
the  IT  artefact  is  the  focus  of  the  research,  Sein  et  al.,  (2011)  suggest  that  alpha  versions  of  the  emerging 
artefact  are  tested  and  evaluated  by  the  practitioners.  After  this  strongly  participatory  process,  a beta  version 
of  the  artefact  is  introduced  to  the  end-users  themselves,  and  again,  the  resulting  evaluations  refine  and  shape 
the  artefact  (Figure  3). 
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Figure  3.  ADR  generic  schema  for  IT-Dominant  Building,  Intervention  and  Evaluation  (Sein  et  al.,  201 1) 

[Figure  3 from  M.  K.  Sein,  O.  Henfridsson,  S.  Purao,  M.  Rossi,  and  R.  Lindgren,  “Action  Design  Research,”  MIS 
Quarterly  (35:1),  201 1,  p.  42.  Copyright  © 2011,  Regents  of  the  University  of  Minnesota.  Reprinted  by  permission] 

The  principles  of  “reciprocal  shaping”,  “mutually  influential  roles”  and  “authentic  and  concurrent 
evaluation”  are  addressed  during  the  BIE  stage.  ADR  suggests  that  the  influence  of  the  IT  artefact  and  the 
organisational  context  are  mutual  and  inseparable.  This  may  result  in  the  original  stance  of  the  researcher 
being  changed  by  the  feedback  from  the  organisation,  and  the  ultimate  version  of  the  IT  artefact  quite 
different  from  the  original  version.  Additionally,  the  researcher  and  the  practitioner  also  influence  each  other. 
The  different  insights  offered  by  these  ADR  team  members  into  the  creation  of  the  artefact,  mean  that  the  end 
result  should  be  more  effective  being  a combination  of  both  theory  and  practice.  It  should  be  noted  that 
individuals  in  ADR  can  perform  multiple  roles,  but  that  these  roles  are  clearly  identified  and  responsibilities 
assigned  in  Stage  1.  It  should  be  clearly  visible  from  figure  3,  the  generic  schema  for  IT-Dominant  BIE,  that 
ADR  does  not  separate  evaluation  from  building.  It  is  hoped  that  any  unanticipated  consequences  are 
surfaced  during  the  evaluation  of  the  alpha  version,  which  then  allows  for  refinement  of  the  artefact,  before 
the  beta  version  is  introduced  to  the  end  user.  The  decisions  regarding  the  shape  of  the  IT  artefact  and  the 
intervention  in  the  real  organisational  setting  should  be  entwined  with  constant  evaluation.  Sein  et  al.  (2011) 
emphasise  that  due  to  the  emergent  nature  of  the  artefact,  the  setting  is  not  controlled.  It  is  their  belief  is  that 
achieving  authenticity  in  a natural  setting  is  more  important  for  ADR.  The  tasks  in  Stage  2 are  therefore  to 
(1)  discover  the  initial  know  ledge -creation  target,  (2)  select  or  customise  the  BIE  form;  (3)  execute  the  BIE 
cycle(s);  and  (4)  assess  the  need  for  additional  cycles  (Sein  et  al.,  201 1,  p43). 

Stage  3,  “Reflection  and  learning”,  is  continuous,  and  runs  alongside  both  Stages  1 and  2.  This  is  where 
the  contributions  to  theory  arise,  and  as  the  artefact  emerges,  adjustments  to  the  research  process  need  to  be 
made  as  the  researcher’s  understanding  of  it  increases.  The  principle  that  is  attached  to  this  stage  is  “guided 
emergence”.  Stage  3 tasks  are  (1)  reflect  on  the  design  and  redesign  during  the  project;  (2)  evaluate 
adherence  to  principles,  and  (3)  analyse  the  intervention  results  according  to  the  state  goals  (Sein  et  al,  2011, 
p44).  The  last  stage  of  ADR  is  the  “Eormalisation  of  learning”.  The  effect  of  the  artefact  on  the  organisation 
should  be  described  as  formalised  learning,  and  it  is  hoped  that  the  changes  that  were  made  during  the  BIE 
stage,  will  now  be  able  to  be  explicated  in  order  to  enhance  the  original  theories  that  were  used  to  create  the 
initial  alpha  version  of  the  artefact.  During  this  stage,  the  principle  of  “generalised  outcomes”  should  be 
applied.  Sein  et  al.  (2011,  p44)  suggest  this  implies  a shift  from  “the  specific  and  unique”  to  “generic  and 
abstract”,  and  that  three  levels  are  involved,  the  “generalisation  of  the  problem  instance,  the  generalisation  of 
the  solution  instance,  and  the  derivation  of  the  design  principles  from  the  design  research  outcomes”.  The 
tasks  that  should  be  accomplished  during  this  stage  are  (1)  abstract  the  learning  into  concepts  for  a class  of 
field  problems;  (2)  share  outcomes  and  assessment  with  practitioners;  (3)  articulate  outcomes  as  design 
principles;  (4)  articulate  learning  in  light  of  theories  selected;  and  (5)  formalise  results  for  dissemination 
(Sein  et  al.,  201 1,  p45). 
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According  to  Sein  et  al.  (2011),  ADR  has  been  developed  with  the  ultimate  goal  of  innovative  design 
knowledge  for  the  particular  class  of  problems.  The  method  acknowledges  that  the  artefact  being  designed 
emerges  from  the  meeting  of  IT  and  the  organisation,  and  that  any  unanticipated  consequences  of 
implementing  the  artefact,  can  be  addressed  immediately,  and  to  the  benefit  of  both  IT,  from  a theory 
perspective,  and  the  organisation,  from  a practice  perspective.  It  is  felt  that  this  approach,  which  combines 
AR  with  DR  will  be  the  best  approach  to  realising  the  objectives  of  this  research,  as  well  as  delivering 
research  that  is  “efficacious  for  solving  the  practical  problems  at  hand”  (Lee,  2010,  p345).  The  performance 
of  the  final  versions  of  the  implemented  modules  will  be  measured  using  the  criteria  of  efficacy,  efficiency 
and  effectiveness  (Figure  4),  as  defined  by  Checkland  & Poulter  (2006). 


MEASURES: 

• Efficacy: 

o Measure  1\  Trainees  able  to  extract  relevant  financial  reports  & accurate  inputs  in  financial 
system 

Too/:  Extracted  financial  reports,  interviews  with  line  managers 
o Measure  2:  Trainee  satisfaction  with  training 
Tool:  Evaluation  form 

• Efficiency: 

Measure:  number  of  freed  trainer  hours 

Tool:  Report  comparing  classroom  & e-learning  trainer  hours 

• Effectiveness: 

o Measure  1\  Improved  Quarterly  financial  reporting  submissions 
Tool:  Report  comparing  quarterly  financial  reports 
o Measure  2:  Improved  financial  job  performance  as  rated  by  Financial  Managers 
Tool:  Interviews 

(For  all  3,  researcher  diary,  notes,  minutes  of  meetings) 

Figure  4.  Proposed  research  measures  to  investigate  the  efficacy,  efficiency  and  effectiveness  of  the  computer-based 

training  for  compulsory  finance  training 


3.  CONCLUSION 

When  implementing  e-learning  in  the  context  of  the  workplace,  it  is  the  personal  experiences  of  the 
participants  that  is  important.  Whether  the  implementation  is  a success  or  not  will  depend  on  the  subjective 
reaction  that  the  trainees  have  when  exposed  to  this  method  of  training.  It  would  seem  that  a good  approach 
to  ensure  that  the  users  of  the  system  are  involved  is  to  use  their  own  feedback  to  adjust  the  modules  so  that 
the  final  experience  is  both  enabling  and  positive.  Having  researched  both  Action  Research  and  Design 
Research  as  possible  research  approaches  that  are  suitable  for  e-learning  research,  it  appears  that  both 
approaches  have  their  critics  and  limitations.  Therefore,  using  a combination  approach,  designated  by  Sein  et 
al.  (2011)  as  Action  Design  Research,  would  seem  advantageous.  It  is  felt  that  this  would  be  the  most 
suitable  way  of  solving  the  specific  training  concern,  whilst  at  the  same  time,  attempting  to  contribute 
towards  improving  knowledge  in  respect  of  e-learning  implementations  in  the  workplace  and  how  to  measure 
the  efficacy,  efficiency  and  effectiveness  of  such  training  (Iversen  et  al.,  2004).  It  is  also  hoped  that  this 
approach  will  address  the  IS  issue  of  being  an  applied  discipline  which  needs  to  consider  assisting 
practitioners,  as  well  as  adding  to  theory  about  IT  and  organisations,  and  the  human  resources  within  the 
organisations  that  use  the  IT. 

As  this  study  attempts  to  contribute  to  practice  by  intervening  in  a real  world  situation,  the  study  is 
limited  to  the  compulsory  training  of  the  workforce  at  a single  institution.  The  training  modules  affected  are 
run  by  the  finance  department  systems  and  user  support  team.  Therefore,  it  would  be  questionable  to  make 
statistical  generalisation  from  the  findings.  However,  according  to  Lee  and  Baskerville  (2003,  p230),  “In 
interpretivism,  a theory’s  pertaining  only  to  the  setting  where  it  was  developed  would  not  detract  from  its 
validity  or  scientific  status”.  Therefore,  as  this  a qualitative  study,  the  objective  is  to  generalise  from  the 
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individual  findings  to  a theory  within  a particular  setting,  rather  than  to  generalise  from  the  sample  to  the 
population  (Conboy  et  ah,  2012;  Lee  & Baskerville,  2003,  Merriam,  1988).  This  study  is  set  within  the 
workforce  of  a particular  organisation  that  employs  a sophisticated,  integrated  financial  system.  Whether  the 
findings  may  be  applied  to  other  organisations,  both  in  Africa  and  abroad,  is  uncertain  due  to  the  composition 
of  the  workforce  and  the  nature  of  the  system,  but  the  descriptive  and  practical  nature  of  the  research  may 
enable  other  researchers  facing  a similar  problematic  situation  within  other  organisations  to  use  the  theory 
and  findings  as  a base  to  develop  additional  theories,  or  to  compare  and  contrast  interpretations. 
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ABSTRACT 

Social  e-learning  is  a process  through  which  learners  achieve  their  learning  goals  via  social  interactions  with  each  other 
by  sharing  knowledge,  skills,  abilities  and  educational  materials.  Adaptive  e-leaming  enables  adaptation  and 
personalization  of  the  learning  process,  based  on  learner  needs,  knowledge,  preferences  and  other  characteristics.  In  this 
paper,  we  present  a case  study  that  analyzes  the  social  interaction  features  of  a social  personalized  adaptive  e-leaming 
system  developed  at  the  University  of  Warwick,  called  Topolor.  We  discuss  the  results  of  a quantitative  case  study  that 
evaluates  the  perceived  usefulness  and  usability.  The  results  demonstrate  a generally  high  level  of  learner  satisfaction 
with  their  learning  experience.  We  extend  the  discussion  of  the  results  to  explore  future  research  directions  and  suggest 
further  improvements  for  the  studied  social  personalized  adaptive  e-learning  system. 


KEYWORDS 

Social  E-Leaming,  Adaptive  E-Learning,  Adaptive  Educational  Hypermedia  System,  Evaluation,  Social  Interaction. 


1.  INTRODUCTION 

The  Internet  and  the  World  Wide  Web  have  made  it  possible  to  provide  new  types  of  learning  environments 
where  learners  can  interact  with  their  peers  and  engage  in  effective  and  attractive  learning  experiences 
(Welsh,  2003).  Social  media  is  comprised  of  Internet-based  applications  that  stand  on  the  ideological  and 
technological  foundations  of  Web  2.0.  These  applications  allow  creation  and  exchange  of  user  generated 
contents  like  never  before  (Kaplan,  2010).  The  increasing  use  of  social  media  and  Web  2.0  tools  as  well  as 
various  social  features  such  as  sharing,  tagging,  rating,  commenting  in  e-learning  systems  can  offer  new 
opportunities  for  communication,  collaboration,  and  active  participation  in  a learning  process  (McLoughlin, 
2011).  Discussions  and  group  work  are  often  integrated  into  collaborative  and  participative  learning  practice, 
providing  a range  of  educational  benefits,  which  are  thoroughly  discussed  in  the  literature  (e.g.  (Hrastinski, 
2009),  (Rovai,  2004)  and  (Woo,  2007)). 

Adaptive  (Educational)  Hypermedia  (A(E)H)  (Brusilovsky,  2004)  is  another  research  direction  that  offers 
improvements  to  the  area  of  e-learning.  AEH  systems  (e.g.  (Cristea,  2007),  (Ghali,  2009)  and  (Eoss,  2009)) 
provide  personalized  learning  experiences  to  individual  learners,  according  to  a range  of  characteristics,  such 
as  learning  goals,  background  knowledge  and  preferences  (Rosmalen,  2009).  The  use  of  adaptation,  along 
with  the  social  affordances  of  Web  2.0  tools,  carries  a great  potential  for  improving  e-learning  systems  and 
learning  experiences.  However,  the  review  of  the  previous  work  indicates  that  current  e-learning  systems 
have  only  marginally  explored  the  integration  of  social  interaction  features  and  adaptation  techniques.  This 
research  intends  to  address  this  gap  by  evaluating  a system  that  was  developed  to  foster  effective  social  and 
adaptive  e-learning  experiences. 

The  aim  of  this  research  is  to  improve  learning  experience  and  learning  outcomes  via  a social  adaptive 
learning  paradigm,  based  on  the  hypothesis  that  extensive  social  features,  personalized  recommendations  and 
Facebook-like  appearance  of  a system,  anticipated  to  make  the  environment  more  familiar  to  learners, 
subsequently  increases  the  usefulness  and  usability  of  the  system.  To  isolate  research  variables,  this  paper 
focuses  exclusively  on  studying  the  usefulness  and  usability  of  the  social  features  in  adaptive  social 
e-learning.  It  is  based  on  our  recent  quantitative  case  study  that  explores  the  use  of  Topolor  (Shi,  et  al., 
2013a;  Shi,  et  al.,  2013b)  - a social  personalized  adaptive  e-learning  system.  The  rest  of  paper  will  review 
the  research  background,  depict  the  social  interaction  tools  in  Topolor,  and  present  the  conducted  experiment 
and  its  results.  And  finally,  the  conclusions  and  the  outline  of  future  work  will  be  described. 
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2.  RESEARCH  BACKGROUND 

Learning  is  inherently  a social  experience.  Social  aspects  of  learning  have  been  emphasized  in  a range  of 
theoretical  frameworks  developed  to  explain  how  people  learn  (e.g.  (Vygotsky,  1978)  and  (Wenger,  2009)). 
Yet,  developing  effective  and  efficient  online  social  learning  environment  remains  an  open  problem.  While 
online  interaction  via  social  networking  services  has  become  widely  accepted  and  heavily  embedded  in 
day-to-day  life,  providing  solutions  that  foster  creation  of  effective  e-learning  spaces  are  not  straightforward. 

Online  interaction  tools  have  also  been  integrated  and  used  in  AEH  (Brusilovsky,  2004).  This  enabled 
adaptation  of  educational  hypertext  to  the  personalized  needs  of  learners  (Brusilovsky,  2000)  in  e-learning 
systems.  Since  early  2000s,  many  AH  frameworks  have  been  proposed,  such  as  AHAM  (Wu,  2002),  the 
Munich  model  (Koch,  2006),  XAHM  (Cannataro,  2002),  LAOS  (Cristea,  2003)  and  GAP  (Knutov,  2008). 
Few  were  later  extended  to  accommodate  some  social  features,  e.g..  Social  LAOS  (SLAOS)  (Ghali,  2009b) 
added  a collaboration  mechanism  into  the  framework,  and  led  to  the  development  of  the  MOT  2.0  system 
(Ghali,  2009c).  It  introduced  social  features  such  as  a chat  tool,  tagging,  rating  and  commenting  on  learning 
content.  However,  while  these  systems  cater  for  personal  needs  within  specific  learning  contexts,  they  are 
often  limited  in  their  strategies  for  adapting  to  social  needs  or  in  their  social  features.  Some  recent  work 
(Simko,  2010)  has  already  proposed  the  need  for  creating  adaptive  and  highly  interactive  integrated  learning 
environments.  However,  their  work  suggests  only  a limited  number  of  mechanisms  for  enabling  social 
interaction.  Hence,  there  is  a gap  for  extending  and  evaluating  social  interaction  tools  in  adaptive  e-learning 
settings.  Additionally,  their  framework  does  not  take  into  account  the  role  of  learner  familiarity  with  other 
social  interaction  tools  from  e-learning  environments  and  social  networking  websites,  such  as  Facebook. 

In  fact,  the  features  that  many  of  the  learners  are  familiar  with  from  social  networking  websites  remain 
missing  from  the  current  e-learning  systems.  For  instance,  sharing  a learning  status,  engaging  in  a simple 
question/answer  exchange  and  sharing  notes  remain  cumbersome  or  impossible  in  many  of  the  available 
systems.  Subsequently,  the  potential  of  adaptation,  recommendation  and  personalization  that  is  based  on  the 
use  of  the  above  social  features  remains  largely  unexplored. 

In  this  paper  we  address  the  above  gap  by  introducing  and  evaluating  a range  of  social  features  previously 
missing  from  the  available  adaptive  e-learning  systems. 


3.  THE  TOPOLOR  SYSTEM 

To  evaluate  our  hypothesis  (SECTION  1),  based  on  our  experiment  of  requirement  analysis  (Shi,  et  al,  2012), 
we  have  developed  the  Topolor  system  built  on  the  Yii  Framework*  and  Bootstrap^.  Topolor  is  made  available 
open  source  and  is  hosted  on  Github^  for  easy  sharing  and  version  control.  It  is  deployed”*  and  used  as  a social 
learning  environment  to  support  some  postgraduate  level  modules  in  the  University  of  Warwick.  The 
registration  for  using  the  Topolor  system  has  been  recently  opened  to  public.  Thus,  a larger  cohort  of  users  is 
expected  in  the  near  future,  providing  opportunities  for  collecting  feedback,  usage  data  and  suggestions  for 
further  improvements. 

The  Topolor  system  mainly  consists  of  three  sub-systems  (Figure  1).  Each  of  these  contains  a set  of 
interaction  features  that  are  generally  referred  to  here  as  the  social  interaction  toolset  (Shi,  et  al.,  2013c).  The 
subsystems  of  Topolor  are  the  following: 

• Topolor-Home  provides  a chronological  list  of  the  learning  statuses  posted  by  individual  learners.  It  also 
provides  access  to  a set  of  interaction  tools  that  encourage  informal  communication  and  collaboration 
such  as  commenting  on,  sharing  and  favoring  of  learning  statuses  (Figure  la). 

• Module  Center  offers  a warehouse  of  online  courses,  as  well  as  provides  adaptive  learning  content 
recommendation,  learning  expert  recommendation,  and  interaction  tools  that  encourage  personalized 
social  e-learning  such  as  sending  messages  to  recommended  learning  experts  (Figure  lb  and  Figure  Id). 


1 http://yiiframework.com 

2 http ://t witter. github .com/bootstrap 

3 https://github.com/aslanshek/topolor 

4 http://www.topolor.com 
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• QSiA  Center  maintains  some  lists  of  questions/answers  related  to  the  learning  contents,  and  provides 
adaptive  question  recommendation,  learning  topic  recommendation,  expert  peer  adaptation  and  social 
interaction  tools  for  discussions  and  practices  (Figure  Ic). 


c.  Q&A  Center  d.  Module  Center  - Quit  for  a Topic 

Figure  1.  User  Interface  of  Topolor 


The  social  interaction  toolset  is  one  of  the  most  important  components  in  the  Topolor  system.  To  provide 
easy  access  to  interaction,  this  toolset  can  be  accessed  from  many  places  in  the  system.  For  instance.  Figure  2 
shows  the  interaction  with  this  toolset  from  the  Topolor-Home  index  page  (Figure  la). 

This  paper  focuses  on  three  social  interaction  tools.  The  status  tool  (Figure  2a)  is  used  to  share  learning 
statuses.  Learners  can  favorite  and  comment  on  each  other’s  posted  learning  statuses;  the  messaging  tool 
(Figure  2b)  is  used  to  send  private  messages  to  others;  and  the  Q&A  tool  (Figure  2c)  is  used  to  ask  and  answer 
questions.  Learners  can  also  use  Q&A  tool  for  discussions. 


MfiKge  QLA  Notr  Todo 


Sutui  Message  NOU  TCKlO 


a.  create  a Status 

sutu5  QSA  Note  Todo 

b.  send  a Message  c.  ask  a Question 


Figure  2.  Social  interaction  toolset:  (a)  learning  status  creation  tool;  (b)  messaging  tool;  (c)  Q&A  tool 


4.  CASE  STUDY  DESIGN 

Topolor  has  been  evaluated  from  various  perspectives  (Shi,  et  al,  2013d).  In  this  section,  we  present  the 
design  of  the  conducted  case  study.  The  case  study  was  comprised  of  three  consecutive  stages;  1)  the 
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experiment  of  using  the  Topolor  system,  2)  the  questionnaire  about  the  usefulness  and  ease  of  use,  and  3)  the 
analysis  of  the  questionnaire  results  and  some  qualitative  feedback  from  the  users. 

4.1  Experiment  Design 

The  experiment  was  conducted  with  the  help  of  21  students  from  the  Department  of  Computer  Science  at  the 
University  of  Warwick,  who  were  registered  for  a 4th  year  MSc  level  module  ‘Dynamic  Web-Based 
Systems’,  and  a lecturer  who  was  leading  this  module.  The  experiment  lasted  for  2 hours,  during  which  the 
students  were  asked  to  learn  a lesson  on  ‘Collaborative  Filtering’  from  the  system  as  well  as  ensuring  to 
perform  specific  tasks  to  familiarize  themselves  with  the  features  related  to  the  provided  social  interaction 
toolset.  The  full  list  of  the  18  tasks  completed  by  the  students  is  listed  in  Table  1. 


Table  1.  Tasks  Performed  by  The  Student 


Status 

Message 

Q&A:  Question 

Q&A:  Answer 

Create  (1) 

Send  (7) 

Create  (9) 

Create  (12) 

Edit  (2) 

Reply  (8) 

Edit  (10) 

Edit  (13) 

Remove  (3) 

Remove  (11) 

Remove  (14) 

Comment  on  (4) 

Share  (15) 

Favorite  (5) 

Favorite  (16) 

Share  (6) 

Add  Tag  (17) 
Edit  Tag  (18) 

4.2  Questionnaire  Design 

Usefulness  and  ease  of  use  are  fundamental  determinants  of  user  acceptance  for  a tool  usage  (Adams,  1992). 
After  running  the  experiment,  each  student  was  asked  to  fill  in  a questionnaire  to  measure  usefulness  and  ease 
of  use  of  the  social  interaction  toolset.  Likert  Scale  (Mclver,  1981)  questions  were  used  to  get  the  feedback  on 
all  available  features,  as  tested  by  tasks  in  Table  1.  The  students  had  to  select  one  of  the  five  responses  for 
usefulness  and  ease  of  use,  respectively,  as  shown  below.  A score  was  assigned  to  each  response,  on  a scale  of 
1-5  as  numbered  below.  After  collecting  the  questionnaires,  the  responses  of  the  students  were  analyzed. 

• usefulness:  1)  very  useless,  2)  useless,  3)  neither  useless  nor  useful,  4)  useful,  and  5)  very  useful. 

• ease  of  use:  1)  very  hard,  2)  hard,  3)  neither  hard  nor  easy,  4)  easy;  and  5)  very  easy. 


5.  ANALYSIS  AND  RESULTS 


Out  of  the  21  students  who  participated  in  the  experiment,  10  students  responded  to  the  optional  questionnaire. 
The  results  extracted  from  the  questionnaire  are  presented  below.  Figure  3 a shows  the  mean  of  the  responses, 
and  Figure  3b  shows  the  standard  deviation.  The  number  of  total  questions  in  the  questionnaire  was  36,  of 
which  18  questions  were  for  testing  usefulness  and  18  for  ease  of  use. 


5 


(1)  (2)  (3)  (4)  (5)  (6)  (7)  (8)  (9)(10)(11X12X13X14X15K16X17X18) 


TASK 

a.  mean  of  the  responses  for  eaeh  task 


1.2 

1 

: 0.8 
0.6 
0.4 
0.2 
0 


(1)  (2)  (3)  (4)  (5)  (6)  (7)  (8)  (9)  (10K11)(12)(13)(14)(15)(16)(17)(18) 
TASK 


b.  Standard  deviation  of  the  responses  for  eaeh  task 


Figure  3.  The  a.  mean  and  b.  standard  deviation  of  the  responses  for  each  task  (Y-axis  presents  rating  scores;  X-axis 
presents  the  task  order;  blue  columns  present  the  usefulness;  red  columns  present  the  ease  of  use) 
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5.1  Usefulness 

The  blue  columns  in  Figure  3a  and  Figure  3b  present  the  usefulness  results.  The  means  of  the  summative 
results  rank  between  3.7  and  4.6.  The  standard  deviations  of  the  overall  results  are  between  0.516  and  0.994. 
All  the  reported  values  of  a mean  are  much  larger  than  3 (the  neutral  response),  suggesting  students’  attitudes 
to  be  generally  positive. 

5.2  Ease  of  Use 

The  red  columns  in  Figure  3a  and  Figure  3b  present  the  ease  of  use  results.  The  means  of  the  overall  results 
rank  between  3.8  and  4.7.  The  standard  deviations  of  the  overall  results  are  between  0.483  and  1.135.  As  all 
the  means  are  greater  than  3,  it  enables  us  to  infer  that  most  of  the  students  found  the  social  interaction  toolset 
to  be  relatively  easy  to  use. 

5.3  Reliability 

We  adopted  Cronbach’s  alpha  to  measure  the  reliability  of  the  test.  According  to  Carmines,  a Cronbach’s 
Alpha  of  0.8  is  considered  as  highly  reliable  (Carmines,  1979).  The  values  of  Cronbach ’s  Alpha  for  each  of  the 
questions  are  shown  in  Table  2.  Both  usefulness  and  ease  of  use  are  considerably  larger  than  0.8,  suggesting  a 
high  level  of  reliability  of  the  results. 


Table  2.  Cronbach’s  Alpha  (Reliability  Statistics) 


Cronbach’s  Alpha 

Cronbach’s  Alpha  Based  on 
Standardized  Items 

Number  of  Items 

Usefulness 
Ease  of  Use 

.934 

.948 

.944 

.957 

18 

18 

6.  DISCUSSION 

In  addition  to  the  questionnaire  data  collected  from  the  students,  we  also  received  some  qualitative  feedback 
from  both  students  and  the  lecturer  of  the  module.  The  general  feedback  was  consistent  with  the  results  of  the 
questionnaire.  However,  the  responses  included  some  specific  suggestions  for  further  improving  some  of  the 
social  interaction  features,  which  ranked  lower  from  the  point  of  usability  and  ease  of  use.  Due  to  the  space 
limitation,  this  paper  focuses  mainly  on  the  quantitative  results  from  the  questionnaire.  However,  some  of  the 
qualitative  feedback  is  discussed  below  as  appropriate. 

Overall,  the  results  from  the  questionnaire  demonstrate  that  the  social  interaction  toolset  is  perceived  to  be 
useful  and  easy  to  use.  83%  of  all  the  features  have  been  rated  by  the  students  as  useful,  and  78%  of  the 
features  as  easy  to  use  (i.e.,  average  mean  > 4).  Consistently  with  the  questionnaire,  qualitative  feedback 
included  a description  of  the  system  as  “similar  to  known  social  networking  sites  (e.g.  Facebook);  fast  and 
responsive”.  Another  respondent  said:  “One  of  the  best  aspects  of  Topolor  is  the  ability  to  interact  with  others 
during  the  process  of  learning”.  We  now  proceed  to  a detailed  discussion  of  the  individual  social  interaction 
tools,  namely  status,  messaging  and  Q&A. 

6.1  Status 

The  questionnaire  results  demonstrate  that  the  feature  (4),  commenting  on  a status,  was  rated  as  the  third  most 
useful  feature  (mean  = 4.5),  and  its  ease  of  use  was  ranked  as  the  fourth  highest  (mean  = 4.5)  among  all  the 
social  interaction  features.  This  result  is  further  supported  by  the  qualitative  feedbacks.  For  example,  one  of 
the  respondents  explicitly  mentioned  that  commenting  on  each  other’s  statuses  was  one  of  his  favorite  features 
for  interacting  with  other  students. 

On  the  other  hand,  (5)  favoring  a status  had  the  lowest  rating  (mean  = 3.7)  on  usefulness.  The  possible 
reason  for  this  could  be  that  the  students  might  not  have  known  what  the  use  of  favoring  a status  was.  We 
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assume  that  it  would  be  necessary  to  develop  a mechanism  for  providing  basic  information  on  less  familiar 
features  such  as  favoring.  Additionally,  wider  use  of  favoring  with  other  features  such  as  questions/answers 
might  also  affect  the  future  patterns  of  use.  Furthermore,  the  possible  reason  for  the  second  lowest  rating  on 
(5)  favoring  a status  for  its  ease  of  use  (mean  = 3.9),  can  be  that  labels  for  favoring/unfavoring  statuses 
became  visible  only  when  the  status  message  was  being  hovered  over.  The  suggested  improvement  would  be 
to  keep  the  labels  and  the  number  of  times  the  statuses  are  favored  always  visible. 

6.2  Messaging 

The  rating  for  (7),  sending  a message,  was,  whilst  high,  the  second  lowest  (mean  = 3.9)  with  regards  to  its  ease 
of  use.  The  possible  reason  for  this  is  the  current  notification  mechanism  for  new  messages.  More  specifically, 
if  a student  was  on  the  messaging  page,  on  receiving  a new  message  a notification  button,  like  ‘You  have  2 
new  messages’,  would  show  up.  Clicking  on  that  button  would  refresh  the  webpage  with  an  AJAX  response 
and  show  the  received  messages  on  the  top  of  the  message  list.  However,  if  the  students  were  on  other 
webpages,  they  wouldn’t  know  whether  they  received  any  new  messages.  Therefore,  the  students  might  have 
had  no  idea  when  and  how  to  start  messaging.  Additionally,  whilst  most  of  the  webpages  in  the  Topolor 
system  provide  at  least  one  tool  for  sending  messages  to  other  students,  such  as  1)  a new  message  box  (Figure 
2b)  and  2)  an  avatar  list  of  the  recommended  learning  peers  (Figure  4a)  that  could  be  clicked  on,  and  then  a 
messaging  box  (Figure  4b)  would  pop-up.  There  are  still  other  webpages  that  did  not  provide  such  tools, 
potentially  affecting  the  results  on  the  ease  of  use  of  sending  messages. 

. .rjv. 

a>exan4r«  Send  meaaaga  to:  jonny 

15  answefsts) 

jonny 

7 antwarsti 

dana 

3 afl»war$ia> 

uaarft  Cancel 

2 answertra^ 

a.  avatar  list  b.  messaging  box 

Figure  4.  a.  Avatar  list  of  recommended  learning  peers  to  send  messages  to;  b.  the  resulting  pop-up  messaging  box 

6.3  Asking  and  Answering  Questions 

The  questionnaire  results  indicated  that  (12)  answering  a question  was  rated  as  the  most  useful  feature  (mean 
= 4.6)  as  well  as  the  easiest  feature  to  use  (mean  = 4.7),  among  all  the  social  interaction  features.  A similar 
result  was  found  from  the  qualitative  feedback,  where  the  way  of  asking  and  answering  questions  was 
explicitly  mentioned  as  favorable.  Furthermore,  (9)  asking  a question  was  rated  very  high  on  the  usefulness 
(mean  = 4.5)  and  ease  of  use  (mean  = 4.5)  too.  Therefore,  we  can  report  with  confidence  that  the  students  were 
very  satisfied  with  features  of  asking  and  answering  questions. 

However,  the  usefulness  of  (18)  editing  the  tags  of  a question  was  rated  as  the  second  lowest  (mean=3.8), 
and  the  usefulness  of  (17)  adding  tags  to  a question  was  rated  as  the  fourth  lowest  (mean=4. 1).  It  seems  that 
tagging  on  questions  was  not  considered  as  useful  as  other  features  of  the  Topolor  system.  We  can  conjecture 
that  when  a student  asked  a question  in  a given  learning  content  area,  the  relation  between  the  question  and  the 
learning  content  would  have  been  automatically  established,  so  that  tagging  the  question  would  not  have 
brought  additional  benefits.  Posting  questions  beyond  the  learning  content  area  would  be  necessary  to  further 
comment  on  this  feature. 

The  ease  of  use  of  the  feature  for  asking/answering  questions,  (10)  editing  a question  was  rated  the  lowest 
(mean  = 3.8).  To  provide  an  attractive  user  experience,  we  used  AJAX  calls  to  implement  the  feature.  For 
example,  when  a student  clicked  on  the  title  or  the  description  of  a question,  it  would  activate  the  HTML 
editing  box;  and  when  the  editing  box  lost  focus,  it  would  be  replaced  by  the  updated  HTML  text.  No  explicit 
buttons  were  provided  to  trigger  editing  actions.  This  might  have  not  attracted  student  attention  to  the 
existence  of  this  functionality.  Even  though  the  style  of  the  mouse  cursor  changes  when  hovering  the  title  or 
the  description  of  a question,  this  hint  might  not  have  been  a clear  enough  indication  to  the  students  about  the 
provided  editing  functionality.  Moreover,  editing  a question  may  require  engagement  with  the  system  over  a 
longer  period  of  time,  so  the  evaluation  of  this  feature  is  to  be  finalized  after  using  the  system  in  the  long  term. 


II 
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7.  CONCLUSION 

In  this  paper,  we  have  1)  presented  the  social  interaction  toolset  of  the  Topolor  system,  2)  reported  a case 
study  with  a quantitative  analysis  on  its  usefulness  and  ease  of  use,  and  3)  discussed  the  results  and  identified 
potential  improvements  of  the  toolset.  The  developed  Topolor  system,  as  shown  in  Table  1,  was  designed  to 
include  a wider  range  of  social  interaction  features  than  previous  AEH  systems.  The  results  of  the  case  study 
show  that  the  social  interaction  toolset  is  found  to  be  useful  and  easy  to  use.  The  overall  attitude  of  the  students 
towards  the  social  interaction  toolset  in  particular  was  very  positive.  The  oral  feedback  was  that  they  would 
have  wanted  to  have  more  lessons  in  it.  Decisive  in  this,  we  believe,  was  the  fact  that  a lot  of  the  social 
features  had  a look  and  feel  familiar  to  them,  similar  to  the  popular  Facebook  environment,  familiarity  that  is 
essential  to  consider  in  designing  such  systems. 

Although  all  of  the  questions  received  positive  responses  from  the  students,  we  are  still  keen  to  improve 
the  social  interaction  toolset  further.  We  reviewed  the  relatively  lower  rated  features  and  discussed  the  possible 
reasons  that  might  have  led  to  lower  ratings.  To  improve  this  toolset,  we  intend  to  conduct  further  research 
based  on  the  presented  results  and  the  discussion,  particularly  in  the  following  directions: 

• Redesign  the  favoring  tool.  First,  we  intend  to  find  a better  way  of  favoring  and  un-favoring  learning 
statuses,  questions,  and  learning  topics.  Second,  we  intend  to  explore  the  use  of  data  on  favored  items  for 
adaptation  and  personalization. 

• Provide  a status-filtering  tool.  We  intend  to  extend  the  existing  mechanism  of  learning  status  sharing  by 
introducing  a support  for  filtering  and  evaluating  its  benefits  for  collaboration.  This  feature  has  the 
potential  to  improve  the  process  of  locating  relevant  statuses  and  communicating  with  learning  peers. 

• Provide  an  auto-tagging  tool.  While  tagging  enables  to  connect  various  concepts  within  the  system, 
students  are  usually  reluctant  to  tag.  Hence,  we  intend  to  enable  Topolor  to  automatically  generate  tags  for 
questions  and  explore  inter-relations. 

• Improve  AJAX  notification  messaging:  We  intend  to  identify  a more  appropriate  mechanism  for  delivering 
system  notifications. 

• Enhance  the  messaging  tool.  We  intend  to  enhance  the  messaging  tool  by  supporting  instant  messaging 
and  appropriate  notification. 

The  development  of  the  next  version  of  the  Topolor  system  has  been  initiated.  Future  experiments  will 
focus  on  evaluation  of  the  new  and  improved  features.  The  future  deployment  of  the  enhanced  system  will 
especially  enable  further  inquiry  into  the  role  of  social  interaction  in  adaptive  e-learning  environments  and 
the  benefits  for  enriching  learning  experiences  and  improving  learning  outcomes. 


ACKNOWLEDGEMENT 

This  research  is  partially  supported  by  the  BlogForever  Project,  funded  by  the  European  Commission  FP7 
(Contract  No.  269963). 


REEERENCES 

Adams,  D.  A.,  Nelson,  R.  R.  and  Todd,  P.  A.,  1992.  Perceived  usefulness,  ease  of  use,  and  usage  of  information 
technology:  a replication.  MIS  Quarterly,  Vol.  16  No.  2,  pp.  227-247. 

Hrastinski,  S.,  2009.  A theory  of  online  learning  as  online  participation.  Computers  & Education,  Vol. 52,  No.l,  pp. 78-82. 
Brusilovsky,  P.,  2000.  Adaptive  hypermedia:  From  intelligent  tutoring  systems  to  Web-based  education.  Intelligent 
Tutoring  Systems,  pp.  1-7. 

Brusilovsky,  P.,  2011.  Adaptive  hypermedia,  User  modeling  and  user-adapted  interaction,  Vol.  11,  No.  1-2,  pp.  87-110. 
Brusilovsky,  P.,  2004.  Adaptive  Educational  Hypermedia:  From  generation  to  generation.  Proceedings  of  4th  Hellenic 
Conference  on  Information  and  Communication  Technologies  in  Education.  Athens,  Greece,  pp.  19-33. 

Cannataro,  M.  et  al.,  2002.  Modeling  adaptive  hypermedia  with  an  object-oriented  approach  and  XML,  In  2nd 
International  Workshop  on  Web  Dynamics. 

Carmines,  E.  and  Zeller,  R.,  1979.  Reliability  and  Validity  Assessment,  Sage,  Beverly  Hill,  CA. 


63 


ISBN:  978-972-8939-88-5  © 2013  lADIS 


Cristea,  A.  and  De  Mooij,  A.,  2003.  LAOS:  Layered  WWW  AHS  authoring  model  and  their  corresponding  algebraic 
operators.  Proceedings  of  WWW’03,  Alternate  Track  on  Education.  Budapest,  Hungary. 

Cristea,  A.,  Smits,  D.  and  de  Bra,  P.,  2007.  Towards  a generic  adaptive  hypermedia  platform:  a conversion  case  study. 
Journal  of  Digital  Information,  Vol.  8,  No.  3. 

Corbett,  A.,  Koedinger,  K.  and  Anderson,  J.,  1997.  Intelligent  tutoring  systems.  Handbook  of  humancomputer 
interaction,  pp.  849-87. 

Ghali,  F.  and  Cristea,  A.,  2009a.  MOT2.0:  A Case  Study  on  the  Usefulness  of  Social  Modeling  for  Personalized  E- 
Leaming  Systems.  Proceedings  of  the  2009  conference  on  Artificial  Intelligence  in  Education:  Building  Learning 
Systems  that  Care:  Erom  Knowledge  Representation  to  Affective  Modelling.  Brighton,  UK,  pp.  333-340. 

Ghali,  F.  and  Cristea,  A.,  2009b.  Social  Reference  Model  for  Adaptive  Web  Learning.  Proceedings  of  International 
Conference  on  Web-based  Learning.  Aachen,  Germany,  pp.  162-171. 

Ghali,  F.  and  Cristea,  A.,  2009c.  Authoring  for  e-learning  2.0:  A case  study.  Proceedings  of  9th  IEEE  International 
Conference  on  Advanced  Learning  Technologies.  Riga,  Latvia,  pp.  170-174. 

Foss,  J.  and  Cristea,  A.,  2009.  Adaptive  Hypermedia  Content  Authoring  using  MOT3.0.  In  ASH:  7th  International 
Workshop  on  Authoring  of  Adaptive  and  Adaptable  Hypermedia  Workshop,  Proceedings  of  the  4th  EC-TEL 
International  Conference,  Nice,  France. 

Haythomthwaite,  C.,  2002.  Building  social  networks  via  computer  networks:  Creating  and  sustaining  distributed  learning 
communities.  Building  virtual  communities:  Learning  and  change  in  cyberspace,  pp.  159-190. 

Kaplan,  A.  and  Haenlein,  M.,  2010.  Users  of  the  world,  unite!  The  challenges  and  opportunities  of  Social  Media. 
Business  horizons,  Vol.  53,  No.  1,  pp.  59-68. 

Knutov,  E.,  2008.  GAF:  Generic  adaptation  framework.  Proceedings  of  the  5th  International  Conference  on  Adaptive 
Hypermedia  and  Adaptive  Web-Based  Systems.  Hannover,  Germany,  pp.  400-404. 

Koch,  N.  and  Wirsing,  M.,  2006.  The  Munich  reference  model  for  adaptive  hypermedia  applications.  Proceedings  of  the 
4th  Adaptive  Hypermedia  and  Adaptive  Web-Based  Systems.  Dublin,  Ireland,  pp.  213-222. 

Mclver  J.  and  Carmines,  E.  G.,  1981.  Unidimensional  Scaling,  Vol.  24,  pp.  40-55. 

McLoughlin,  C.  and  Lee,  M.,  2011.  Pedagogy  2.0:  Critical  challenges  and  responses  to  Web  2.0  and  social  software  in 
tertiary  teaching.  In  M.  J.  W.  Lee  and  C.  McLoughlin  (Eds.),  Web  2.0-based  e-learning:  Applying  social  informatics 
for  tertiary  teaching,  pp.  43-69. 

Rosmalen,  Van,  et  al.,  2009.  Authoring  a full  life  cycle  model  in  standards-based,  adaptive  e-leaming.  Proceedings  of  the 
9th  IEEE  International  Conference  on  Advanced  Learning  Technologies.  Riga,  Latvia,  pp.  170-174. 

Rovai,  A.  P.,  2004.  A constructivist  approach  to  online  college  learning.  The  Internet  and  Higher  Education,  Vol.  7,  No. 
2,  pp.  79-93. 

Scotton,  J.,  et  ah,  2009.  A Case  Study  on  Merging  Strategies  for  Authoring  QoE-based  Adaptive  Hypermedia.  In  ASH: 
7 th  International  Workshop  on  Authoring  of  Adaptive  and  Adaptable  Hypermedia  Workshop,  Proceedings  of  the  4th 
EC-TEL  International  Conference,  Nice,  France. 

Shi,  L.,  AL-Qudah,  D.,  Cristea,  A.  L,  2012.  Exploring  Participatory  Design  for  SNS-based  AEH  Systems.  Proceedings  of 
1 1th  lADIS  International  Conference  WWW/INTERNE.  Madrid,  Span,  pp.  242-249. 

Shi,  L.,  et  al.,  2013a.  Topolor:  A Social  Personalized  Adaptive  E-Learning  System.  Proceedings  of  2Ist  Conference  on 
User  Modeling,  Adaptation  and  Personalization.  Rome,  Italy,  LNCS,  vol.  7899,  pp.  338-340. 

Shi,  L.,  Gkotsis,  G.,  Stepanyan,  K.,  Al-Qudah,  D.  A.,  Cristea,  A.  I.  2013b.  Social  Personalize  Adaptive  E-Learning 
Environment  - Topolor:  Implementation  and  Evaluation.  Proceedings  of  1 6th  International  Conference  on  Artificial 
Intelligence  in  Education.  Memphis,  USA,  LNCS,  vol.  7926,  pp.  708-711. 

Shi,  L.,  et  ah,  2013c.  Designing  Social  Personalized  Adaptive  E-Learning.  Proceedings  of  the  I8th  Annual  Conference 
on  Innovation  and  Technology  in  Computer  Science  Education.  Canterbury,  UK  (in  press). 

Shi,  L.,  et  al.,  2013d,  Towards  Understanding  Learning  Behavior  Patterns  in  Social  Adaptive  Personalized  E-Leaming 
Systems,  Proceedings  of  19th  Americas  Conference  on  Information  Systems,  Chicago,  Illinois,  USA  (in  press) 

Simko,  M.,  Barla,  M.  and  Bielikova,  M.,  2010.  ALEF:  A framework  for  adaptive  web-based  learning  2.0,  Key 
Competencies  in  the  Knowledge  Society,  pp.  367-378. 

Vygotsky,  L.,  1978.  Mind  in  society. 

Welsh,  E.,  et  ah,  2003.  E-leaming:  emerging  uses,  empirical  results  and  future  directions.  International  Journal  of 
Training  and  Development,  Vol.  7,  No.  4,  pp.  245-258. 

Wenger,  E.,  2009.  A social  theory  of  learning.  Contemporary  theories  of  learning:  learning  theorists  in  their  own  words, 
pp.  209-218. 

Woo,  Y.  and  Reeves,  T.,  2007.  Meaningful  interaction  in  web-based  learning:  A social  constmctivist  interpretation.  The 
Internet  and  Higher  Education,  Wo\.  10,  No.  1,  PP.  15-25. 

Wu,  H.,  2002.  A reference  architecture  for  Adaptive  Hypermedia  Applications  (Doctoral  dissertation,  TUE). 


64 


I ADIS  International  Conference  e-Learning  2013 


TRAINING  EDUCATORS: 

PLAN  FOR  REPLICATING  THE  EXPERIENCE 


Ulanbek  Mambetakunov^  and  Marina  Ribaudo^ 

‘ Bishkek  Academy  of  Finance  and  Economics,  Kyrgyzstan 
^DIBRIS,  University  of  Genova,  Italy 


ABSTRACT 

This  paper  reports  on  a short  survey  and  a training  course  offered  to  the  faculty  members  of  the  University  of  Genova 
with  the  aim  of  driving  the  users  of  the  university  Learning  Management  System  in  the  transition  from  Moodle  1.9  to 
Moodle  2,  transition  that  took  place  in  August  2012.  The  survey  has  been  administered  to  lectures  and  staff  to  understand 
their  needs  and  gather  subscriptions  before  the  actual  starting  of  the  training  course.  A similar  process  is  now  in  progress 
in  the  context  of  the  Bishkek  Academy  of  Finance  and  Economics  (Kyrgyzstan),  where  the  same  survey  has  been 
recently  delivered.  Results  obtained  in  both  institutions  are  presented.  Indeed,  although  we  are  aware  of  the  fact  that  these 
two  universities  represent  two  different  contexts,  being  geographically  far,  different  in  size,  and  with  different  diffusion 
of  ICT,  we  think  we  can  take  advantage  of  the  experience  of  the  University  of  Genova  to  transfer  some  practices  on  the 
organization  and  delivery  of  training  courses  in  the  Bishkek  Academy  of  Finance  and  Economics. 


KEYWORDS 

Moodle,  training  educators,  blended  learning,  LMS. 


1.  INTRODUCTION 

In  the  last  two  decades  schools,  universities,  and  companies  had  to  prepare  for  transformation  and  future 
learning  in  the  knowledge  and  networked  society.  In  the  context  of  higher  education,  many  universities  have 
posed  a great  emphasis  on  promoting  ICT  solutions  for  education  practices  and  one  of  the  most  popular 
choice  has  been  the  introduction  of  Learning  Management  Systems  (LMS)  to  facilitate  anytime,  anywhere 
access  to  learning  content  and  course  administration,  with  different  levels  of  success  (Black  et  ah,  2007; 
Gautreau,  2011;  Janossy,  2008). 

Among  LMSs,  Moodle  (Cole  and  Foster,  2007)  is  probably  the  most  popular  alternative  when  deciding  to 
adopt  an  open  source  solution,  as  witnessed  by  the  wide  community  of  users  and  by  the  active  community  of 
developers.  Many  universities  are  indeed  offering  Moodle  installations  to  their  instructors  and  learners  (see 
the  official  Moodle  site  https://moodle.org/stats/  for  some  statistics).  However,  since  it  is  easier  to  build 
learning  infrastructures  than  to  change  one’s  teaching  style,  experience  shows  that  many  Moodle  instances 
have  been  (and  still  are)  mostly  used  as  content  repositories  despite  current  trends  in  education  suggest  to 
take  advantage  of  the  array  of  available  on-line  tools  to  promote  a collaborative  learning  approach  with  new 
roles  for  teachers  and  students  (Downes,  2005;  Anderson,  2007). 

Embedding  ICT  in  education  is  a slow  but  constant  process  and  the  universities  can  play  different  roles. 
From  a research  perspective,  experts  in  education  should  develop  new  solutions  and  pedagogical  models  for 
future  teachers,  including  strategies  for  engaging  students  of  the  21st  century  (Prensky,  2001).  Technologists 
should  enhance  traditional  LMSs  with  new  learning  environment  prototypes,  open  access  repositories,  social 
software  based  on  the  Web  2.0  paradigm  to  promote  learners  communication,  peers  collaboration,  active 
participation  (Mott,  2010). 

From  an  educational  perspective,  research  results  in  this  field  should  also  be  internally  disseminated  and 
adopted  to  “modernize”  current  university  teaching  practices.  Faculty  members  are  rich  sources  of  content  in 
different  areas,  but  they  have  different  teaching  philosophy  and  they  are  not  necessarily  instructional 
designers  or  media  developers.  Therefore  they  need  help  to  transform  their  domain  knowledge  into  digital 
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resources  to  be  used  on-line,  possibly  without  excessive  preparation  time.  They  also  need  to  be  trained  to 
take  advantage  of  technology  when  used  as  a tool  for  education  (Elgor,  2005;  Kampov-Polevoi,  2010). 

When  possible,  universities  or  single  departments  have  invested  into  e-learning  or  media  centers  to 
provide  this  kind  of  support  trying  to  engage  faculty  members  and  students  in  a mix  of  face-to-face  and  on- 
line sessions.  This  is  indeed  the  current  situation  both  at  the  University  of  Genova  (UNIGE)  and  at  the 
Bishkek  Academy  of  Einance  and  Economics  (BAEE),  although  with  different  diffusion,  with  the  experience 
of  UNIGE  being  more  mature  with  respect  to  the  current  situation  in  BAEE.  In  this  work  we  first  introduce 
the  two  contexts  (Section  2)  and  then  compare  the  results  of  the  surveys  which  have  been  delivered  within 
the  two  communities  (Section  3).  We  then  briefly  present  the  training  course  which  has  already  been  offered 
at  UNIGE  (Section  4)  and  a plan  for  a similar  course  to  be  offered  at  BAEE  trying  to  take  advantage  from  the 
lesson  already  learned  in  UNIGE  (Section  5). 


2.  THE  TWO  CONTEXTS 


2.1  University  of  Genova 

The  University  of  Genova,  Italy,  is  a traditional  university,  delivering  face-to-face  courses.  The  educational 
offer  ranges  from  sciences  to  humanities,  from  medicine  to  engineering,  from  economics  to  education. 
Starting  from  2004  UNIGE  has  offered  to  its  students,  faculty  members,  and  staff  a Moodle -based  software 
platform  which  is  daily  used,  mostly  following  a blended  learning  approach  (Cerioli  et  ah,  2011).  The  users 
of  the  LMS  are  of  several  types: 

1 . Faculty  members,  who  extend  their  teaching  practices  with  on-line  sessions. 

2.  Students,  who  find  on-line  courses  and  also  other  services. 

3.  Referents,  i.e.  members  of  the  administrative  staff  involved  into  the  administration  and  management 
of  organizational  procedures  related  to  education. 

4.  Technical  staff,  i.e.  two  technicians  responsible  for  its  correct  functioning. 

5.  A small  Board  of  faculty  members  with  different  skills,  who  supervise  the  whole  project,  providing 
guidelines  and  directions  for  future  development. 


Table  1.  Size  of  the  on-line  community  in  UNIGE 


On-line  community 
(21/04/2013) 

University 

(31/12/2012) 

Referents 

37 

- 

Teachers 

916 

1323  (69.23%) 

Students 

28236 

32393  (87.16%) 

Courses 

1499 

- 

Table  1 shows  some  absolute  numbers  on  the  size  of  the  LMS  community;  in  the  third  column  the 
number  of  faculty  members  and  students  of  the  university  are  given  to  provide  some  percentage  values. 
According  to  these  values,  7 lecturers  in  10  manage  one  or  more  on-line  courses,  and  more  than  8 students  in 
10  access  to  the  LMS. 

Although  these  numbers  are  significant,  the  “quality”  of  the  average  exploitation  of  the  available  software 
tools  for  education  is  generally  poor,  and  therefore  the  university  has  organized  in  the  past  methodological 
courses  on  Instructional  Design  (Puddu  et  ah,  2008),  based  on  a variant  of  the  ADDIE  (Analysis,  Design, 
Development,  Implementation,  Evaluation)  model;  see  (Molenda,  2003)  for  a discussion  on  the  origin  of  the 
term  ADDIE  and  http://www.instructionaldesign.org/  for  links  to  material  on  Instructional  Design.  Technical 
courses  on  the  use  of  Moodle's  functions  have  been  organized  as  well  to  empower  teachers  in  the 
management  of  their  on-line  courses.  Anonymous  surveys  are  also  periodically  launched  to  assess  the 
acceptance  of  the  platform  or  for  organizational  purposes.  The  last  survey  has  been  administered  in  Eebruary- 
March  2012,  to  promote  a forthcoming  technical  training  course  on  Moodle  2 among  faculty  members  and 
referents,  before  the  software  upgrade  planned  for  August  2012.  The  survey  will  be  presented  in  Section  3. 


66 


I ADIS  International  Conference  e-Learning  2013 


2.2  Bishkek  Academy  of  Finance  and  Economics 

Kyrgyzstan  is  a landlocked  republic  in  the  eastern  part  of  Central  Asia.  Bishkek  is  the  capital  and  largest  city. 
The  country  counts  52  universities  and  institutes:  31  state  and  21  private.  In  the  capital  there  are  21  state  and 
9 private  universities.  The  total  number  of  students  across  the  country  currently  stands  at  230,000  people  and 
the  literacy  rate  is  98.7%.  Bishkek  Academy  of  Finance  and  Economics  is  a private  academy  offering  mostly 
scientific  and  economic  courses;  the  academy  counts  53  faculty  members  and  1200  students. 

BAFE  started  offering  its  Moodle  1.9  platform  in  2011;  the  users  of  the  LMS  are  faculty  members  and 
first  grade  students.  Like  in  UNIGE,  the  platform  is  administered  by  a small  technical  staff  coordinated  by 
the  chief  of  the  Fundamental  Sciences  Department. 

The  survey  administered  in  BAFE,  described  in  detail  in  Section  3,  proposed  also  some  additional 
questions  (see  Table  2)  to  better  understand  the  familiarity  with  technology  for  this  sample  of  users.  Question 
SI  checks  the  frequency  of  access  to  Internet  and  more  than  one  third  of  the  respondents  declare  they  can 
access  only  once  per  month.  Question  S2  investigates  the  type  of  equipment  used  to  access  to  Internet:  25 
respondents  use  a personal  computer,  but  often  this  is  located  in  one  department  room,  and  shared  among 
several  lecturers.  The  third  question,  S3,  asks  about  access  through  mobile  devices:  Only  2 in  40  respondents 
answer  “Yes”.  Indeed,  access  to  Internet  through  mobile  devices  for  Kyrgyzstan's  teachers  is  still  too 
expensive,  and  therefore  95%  of  the  respondents  gave  a negative  answer. 


Table  2.  Additional  questions  for  the  survey  in  BAFE 


Question 

Answers 

SI.  How  often  do  you  access  to  the  Internet? 

- Every  day 

- Twice  a week 

- Once  per  month 

- Other 

14  (35.0%) 
10  (25.0%) 
16  (40.0%) 

S2.  Which  electronic  equipment  do  you  use  to 
access  to  Internet? 

- Personal  computer 

- Notebook 

- Mobile  phone 

- iPad 

25  (62.5%) 
14  (35.0%) 
1 (02.5%) 

S3.  Can  you  access  to  Internet  through  a 
mobile  phone? 

- Yes 
-No 

2 (05.0%) 
38  (95.0%) 

Despite  the  fact  that  Kyrgyzstan  has  now  5 Internet  providers,  residential  access  to  the  network  is 
expensive  too  and  only  a small  proportion  of  families  can  afford  to  have  Internet  at  home.  Therefore  also 
students  have  difficulties  in  accessing  on-line  services.  Moreover,  BAFE  has  students  coming  from  rural 
areas  and  many  of  them  do  not  have  basic  skills  for  using  computer  programs.  The  academy  provides  access 
to  computer  labs  with  modern  equipments  and  Internet  access  for  the  lectures  but  it  cannot  offer  access  to  the 
laboratories  for  self-study. 

Given  these  premises,  it  should  be  clear  that  the  current  situation  in  BAFE  is  different  from  that  in 
UNIGE:  if  lectures  can  use  Internet  only  once  per  month  and  students  cannot  access  as  they  would  like,  the 
use  of  Moodle  cannot  go  quickly  formal.  However,  the  adoption  of  the  LMS  has  started  and,  despite  all  the 
difficulties,  the  process  is  not  reversible. 


3.  THE  SURVEY 

The  survey,  which  has  been  administered  in  both  institutions,  is  composed  of  10  common  questions. 
Although  we  are  aware  that  UNIGE  and  BAFE  are  very  different  in  size,  place,  and  adoption  of  ICT,  we  will 
present  the  results  in  the  same  figures  to  save  space  and  discuss  the  outcomes  in  parallel. 

At  UNIGE,  the  survey  has  been  advertised  through  the  university  mailing  list,  and  it  has  been 
administered  for  three  weeks  using  the  feedback  Moodle  module,  in  February-March  2012.  174  employees, 
including  faculty  members  and  administrative  staff  involved  in  the  use  of  the  LMS,  answered  to  the  proposed 
questions,  on  a voluntary  base. 
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At  BAFE,  the  survey  has  been  copied  on  paper  and  distributed  into  the  post  office  box  of  faculty 
members,  with  a letter  of  the  Rector.  The  survey  has  been  motivated  by  the  need  of  understanding  the  use  of 
Moodle  in  the  academy,  one  year  after  its  introduction.  40  faculty  members  answered  to  the  survey  that 
means  a response  rate  of  75%  (since  the  academy  has  53  faculty  members).  This  high  response  rate  is 
probably  due  to  the  fact  that  letter  of  the  Rector  made  the  survey  official. 

Question  Q1  is  about  the  use  of  Moodle  and  Figure  1 shows  the  responses  in  absolute  numbers  (in  this 
and  in  the  following  figures,  the  caption  reports  the  text  of  the  question). 


No  Rarely  Yes.  at  least  once  Yes.  every  day 

perweek 


Figure  1.  Answers  to  question  Q1  "Do  you  use  Moodle?" 

Q2  is  a multiple  choice  question,  and  we  asked  respondents  who  use  Moodle  (138  in  UNIGE  and  32  in 
BAFE)  to  indicate  the  resources  and  activities  most  commonly  used.  As  expected,  the  content  repository 
function  appears  to  be  the  most  popular  in  both  universities,  as  shown  in  Figure  2.  Indeed,  university  teachers 
are  not  necessarily  instructional  design  experts  and  therefore  they  tend  to  shift  their  teaching  practices  from 
the  class  to  the  on-line  platform.  This  means  primarily  a transformation  of  their  lectures  into  digital  learning 
objects.  In  BAFE,  assignments  and  questionnaires  for  self-assessment  and  exams  get  a score  comparable  to 
that  of  file  upload.  Lectures  who  use  Moodle  are  trying  to  automate  content  delivery  but  also  assignments 
and  exams  that  most  likely  occur  within  the  academy  computer  labs,  during  working/lecture  hours. 

On  the  other  hand,  the  lack  of  Internet  connection  at  home,  as  highlighted  in  Section  2,  does  not  provide 
many  opportunities  for  on-line  asynchronous  communication  with  students  in  BAFE,  and  therefore  the  use  of 
forum  is  not  much  developed,  while  it  results  as  the  second  mostly  used  tool  in  UNIGE.  Wiki  is  not  selected 
by  any  respondent  in  BAFE,  probably  for  the  same  reason  or  because  the  respondents  do  not  really  know  the 
meaning/potentiality  of  such  a tool. 

The  satisfaction  level  (Q3)  for  those  who  use  Moodle  turns  out  to  be  moderate  in  UNIGE:  45  respondents 
affirm  to  be  satisfied,  answering  “Yes,  and  I could  not  cope  without  it”,  while  the  second  answer  “Yes,  but  it 
is  too  time  consuming”  is  chosen  by  16.  There  is  of  course  room  for  improvement  to  capture  those 
respondents  saying  they  are  “Enough”  satisfied  (44)  and  those  answering  they  are  not  (“Little”  or  “No”  for 
8).  In  BAFE  the  answer  “Enough”  gets  the  highest  score  (15)  followed  by  “Little”  (5)  and  “Yes,  but  it  is  too 
time  consuming”  (5),  thus  suggesting  room  for  improvement  also  in  this  case.  Figure  3 shows  the  answers  to 
this  question,  in  percentage  values. 


68 


lADIS  International  Conference  e-Learning  2013 


80,00% 
70,00%  - 
60,00%  - 
50,00%  - 
40,00%  - 
30,00% 
20,00%  - 
10,00%  - 
0,00% 


□ UnivA 
oAcadB 


In  . ■ . 


Figure  2.  Answers  to  question  Q2  ‘‘If  you  answered  yes  to  Ql,  which  are  the  platform  functions  you  use  more?” 
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Figure  3.  Answers  to  question  Q3  “If  you  answered  yes  to  QI,  are  you  satisfied  ofMoodle?” 

Q4  is  an  open-ended  question  aiming  at  understanding  whether  the  participants  use  other  external  tools 
for  teaching.  All  BAFE's  respondents  declare  they  do  not  use  other  on-line  tools  and  the  situation  is  similar  in 
UNIGE  for  the  majority  of  participants;  among  the  few  answers  we  recall  Google  Docs  (cited  14  times)  and 
personal  websites  with  teaching  material  (6  times). 

Also  Q5  is  an  open-ended  question  in  which  respondents  should  indicate  the  tools  used  for  the 
development  of  their  teaching  material.  In  the  case  of  UNIGE,  Microsoft  PowerPoint  is  the  leader  being 
mentioned  in  81  responses.  Those  who  use  PowerPoint  also  mention  Word  (10)  and/or  Excel  (5).  Other  tools 
which  received  some  scores  are  Adobe  Acrobat  (11),  LaTex  (8)  and  the  OpenOffice  suite  (5).  Specialized 
softwares  such  as  WebMatrix,  Adobe  Creative  Studio,  Articulate  Studio,  HotPotato,  and  others,  got  a single 
point  each.  In  the  case  of  B APE,  only  Microsoft  Office  is  reported  (with  Word,  PowerPoint,  and  Excel). 

Q6  asks  to  list  the  information  sources  considered  relevant  in  the  construction  of  teaching  material. 
Scientific  papers  and  journals,  digital  material  accompanying  textbooks  and  slides  of  the  colleagues  are  the 
most  cited  sources  for  respondents  from  both  institutions.  The  answers  to  this  question  show  the  differences 
among  the  various  disciplines.  Querying  of  medical  online  databases,  scientific  literature,  online  consultation 
of  laws  and  regulations,  newspapers  and  online  dictionaries  are  responses  dictated  by  the  disciplines  thought 
by  the  respondents.  Wikipedia  is  mentioned  only  twice  (and  only  in  UNIGE),  probably  because  its  content  is 
considered  unreliable  by  the  faculty  members  of  both  samples. 

The  last  four  questions  are  related  to  the  organization  of  the  training  course  itself:  129  respondents  in 
UNIGE  claim  to  be  interested  in  attending  it,  declaring  to  be  mostly  interested  into  Moodle  2 novelties  (110), 
followed  by  the  always  present  repository  functionality  (87),  by  online  questionnaires  (76)  and  by  the  Lesson 
module  (75).  In  the  case  of  BALE,  1 1 respondents  answered  “I  would  be  interested  but  I have  no  time  to 
attend  it”  and  2 said  definitely  “No”.  Among  those  who  declare  an  interest  in  attending  the  course,  the  most 
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required  functions  result  file  upload  (13),  assignment  (14),  questionnaire  (14)  and  lesson  (11);  only  one 
respondent  is  willing  to  know  more  about  the  wiki  function.  In  BAFE's  survey  question  Q8  did  not  propose 
Moodle  2 novelties  among  the  possible  choices.  The  adoption  of  Moodle  1.9  is  still  recent  in  the  academy, 
and  therefore  we  thought  that  suggesting  also  a software  upgrade  could  have  confused  (and  probably 
frustrated)  the  users. 

In  question  Q9  participants  had  to  evaluate  their  skills  in  the  use  of  Moodle  and  identify  themselves  as 
“Beginner”,  i.e.,  user  who  had  never  or  very  little  used  the  platform,  “Intermediate”,  i.e.,  user  who  already 
had  some  experience,  or  “Expert”.  Eigure  4 shows  how  respondents  self-assessed  themselves.  The  answers  to 
this  question  confirm  that  the  experience  in  UNIGE  is  more  mature  with  respect  to  BATE  where  the  majority 
of  the  sample  (39)  selected  the  option  “Beginner”  and  no  one  selected  the  option  “Expert”. 


Figure  4.  Answers  to  question  Q9  "How  do  you  evaluate  your  knowledge  of  the  platform?” 


4.  THE  TRAINING  COURSE  IN  UNIGE 

In  May-June  2012  the  technical  course  started  in  UNIGE.  Eollowed  on  a voluntary  base,  the  course  has 
offered  to  the  learners  the  possibility  to  test  in  advance  the  new  functionalities  of  Moodle  2.  It  has  also 
provided  to  the  technical  staff  the  opportunity  of  checking  the  proper  functioning  of  the  prototype  with  small 
groups  of  participants,  before  the  full  scale  adoption  for  the  whole  community  of  users. 

Participants  have  been  organized  in  groups,  formed  on  the  basis  of  their  self-reported  skills.  Another 
indication  for  groups  formation  has  been  the  venue  of  the  practical  lessons  which  have  been  replicated  in 
different  areas  of  the  town.  133  users  took  part  to  this  phase  and  where  split  into  four  groups,  for  as  many 
parallel  courses  (two  Beginner  and  two  Intermediate).  One  tutor,  in  fact  a member  of  the  board  “Expert”  in 
the  use  of  Moodle  2,  was  assigned  to  each  group  together  with  a member  of  the  technical  staff. 

Lessons  followed  a blended  learning  approach:  Eace-to-face  meetings  have  been  organized,  supported  by 
on-line  tutoring  through  the  Moodle  2 platform  itself.  The  first  plenary  lecture,  organized  for  all  participants, 
introduced  the  use  of  ICT  and  Web  2.0  in  education  and  some  basics  on  Instructional  Design.  Then,  for  each 
group,  technical  lectures  took  place  in  computer  labs.  Table  3 shows  the  course  syllabus,  with  the  number  of 
hours  dedicated  to  beginner  and  intermediate  participants. 

For  each  group  a working  space  in  a new  Moodle  2 installation  has  been  prepared.  The  Moodle  2 space 
was  organized  by  topics  and  each  participant  has  been  asked  to  work  in  a single  block  (a  topic  within  a 
Moodle  course),  covering  the  teacher  role.  With  this  choice,  each  participant  could  see  the  work  of  the  peers. 

No  ad-hoc  material  has  been  prepared  but  on-line  videos  and  documents  have  been  used  as  teaching 
material,  in  addition  to  the  resources  available  at  the  official  website  http://docs.moodle.org. 

Working  in  a Moodle  2 course  with  all  participants  acting  as  teachers  was  not  easy  since  they  all  had  the 
privileges  to  update  course  settings  and  to  add/edit/delete  course  blocks,  resources  and  activities.  In  addition, 
some  configurations,  such  as  the  notification  of  the  delivery  of  assignments,  caused  an  unusual  number  of 
messages  that  generally  does  not  occur  in  a “normal”  course  with  only  one  or  few  teachers.  However,  also 
these  unintentional  errors  have  increased  the  knowledge  of  the  platform,  encouraging  the  learners  to  carefully 
check  the  configuration  parameters  of  each  new  element  added  into  the  on-line  space. 
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Table  3.  Training  course  syllabus 


Content 

Beginner 

Intermediate 

Plenary  lecture 

ICT  and  Web2.0  in  education.  Instructional  Design. 

2h 

2h 

First  practical 

Course  settings,  users  and  roles  management,  forum, 
Moodle  web  page,  file  and  directory  management, 
interaction  with  external  repositories. 

6h 

2h 

Second  practical 

Assignment,  poll,  reservation,  questionnaire,  feedback. 

6h 

2h 

Third  practical 

Gradebook,  groups,  conditional  activities. 

2h 

Particular  emphasis  has  been  posed  on  the  Moodle  2 new  file  management,  a feature  that  has  undergone  a 
major  change  in  the  new  version  of  the  software.  The  File  Picker  offers  a new  solution  to  file  management, 
but  the  renewed  interface  has  caused  difficulties  to  most  of  the  users.  In  addition,  the  possibility  of 
associating  meta  data  with  files,  including  the  license  of  use,  has  highlighted  a lack  of  knowledge  of  this 
subject  among  all  learners.  Fortunately,  meta  data  can  be  omitted,  being  an  optional  feature. 

Conditional  activities,  now  present  in  the  default  configuration,  have  been  another  interesting  novelty. 
Conditional  activities  offer  a sophisticated  tool  to  build  different  educational  paths,  dependent  from  the 
fulfilment  of  multiple  conditions;  none  of  the  course  participants  had  never  used  them  before. 

The  new  graphical  interface,  including  the  drag&drop  option,  has  captured  some  interest.  The  possibility 
of  hiding  blocks  in  the  Dock  (an  area  on  the  left  side  of  the  page  where  blocks  are  minimized)  has  created 
problems  to  the  users  who  have  inadvertently  hidden  a block  in  the  Dock  and  did  not  know  how  to  restore  it. 

The  Lesson  module,  even  if  requested  in  the  initial  survey,  has  not  been  seen  in  detail.  In  fact,  with  the 
introduction  of  conditional  activities,  personalized  learning  paths  can  be  obtained  without  recurring  to  the 
Lesson  control  flow.  The  Lesson  module  seemed  too  sophisticated  for  the  needs  of  the  learners  and  therefore, 
although  initially  planned,  was  put  aside. 

As  a curiosity,  a moment  of  “confusion”  occurred  when  new  links  appeared  as  the  content  of  the  course 
started  shaping.  This  was  due  to  the  Autolink  feature,  activated  by  default  on  the  site.  This  feature  creates 
automatic  links  matching  words  or  phrases  used  in  a course  (in  web  pages,  forum  messages,  etc..)  with 
corresponding  names  in  glossary  and  database  entries,  activities  and  resources.  In  a course  where  many  users 
can  create  resources  and  activities,  often  using  the  same  words  (“quiz  test”,  “test  message”,  “test  text”,  etc.), 
an  unusual  hypertext  suddenly  appeared.  Luckily,  Autolink  can  be  disabled,  as  it  was  promptly  done. 

All  the  four  parallel  courses  reached  their  end,  with  less  difficulties  and  more  work  done  - as  expected  - 
for  the  intermediate  users  who  had  already  some  experience  in  the  use  of  the  previous  version  of  the  LMS. 
Even  in  the  absence  of  a course  feedback,  the  first  impression  is  that  learners  enjoyed  Moodle  2,  as  witnessed 
by  the  enthusiasm  demonstrated  during  the  practicals  and  written  in  a few  private  emails. 

Formal  feedbacks  are  planned  for  the  end  of  this  academic  year.  This  will  be  the  real  test  to  understand 
whether  the  initial  enthusiasm  of  early  course  participants  is  confirmed  by  the  majority  of  colleagues  and  - 
even  more  important  - by  the  community  of  students. 


5.  PLAN  FOR  THE  TRAINING  COURSE  IN  BAEE 

Despite  the  difficulties  already  discussed  in  Section  3,  especially  the  lack  of  computers  for  personal  work,  the 
few  opportunities  for  accessing  the  Internet  outside  working  hours,  the  lack  of  specialists  who  can  introduce 
the  use  of  ICT/Moodle  for  education,  and  also  the  lack  of  time,  the  adoption  of  the  LMS  has  started  in  BAFE. 
But,  to  overcome  the  initial  experimental  phase  and  turn  it  into  a reality,  both  educators  and  students  need  to 
be  trained. 

All  respondents  to  the  survey,  except  one,  self-assessed  themselves  as  “Beginner”,  and  therefore  a single 
training  course,  with  the  same  syllabus  for  all  participants,  is  planned.  The  training  course  cannot  be  centred 
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on  the  mere  transfer  of  technological  knowledge  but  should  first  introduce  some  basic  knowledge  on 
Instructional  Design.  University  teachers,  in  fact,  rarely  come  into  contact  with  educational  models  but  in 
general  they  refine  a personal  style  in  the  design  and  the  management  of  the  teaching/leaming  process  mainly 
based  on  in-presence  lessons.  However,  this  approach  cannot  work  in  technology  enhanced  learning,  which 
depends  on  instructional  design  no  matter  which  is  the  chosen  learning  approach  - content  driven  learning, 
blended  learning,  collaborative  learning. 

Without  becoming  experts,  teachers  should  be  aware  of  the  design,  development  and  management  of 
online  activities  and  therefore  we  suggest  to  start  the  course  with  an  introduction  on  this  subject.  After  that, 
technical  lectures  on  the  use  of  Moodle  should  be  delivered  in  a computer  class  using  an  interactive 
whiteboard  or  video-projector,  and  followed  by  on-line  support  in  a blended  learning  style.  For  each  tool, 
participants  should  be  encouraged  to  put  their  hands-on  for  practical  experience.  In  the  first  edition  of  the 
course  we  do  not  think  that  it  is  necessary  to  introduce  all  Moodle  functions,  but  only  a subset  of  them. 
Following  the  experience  of  UNIGE,  we  plan  a course  of  14  hours,  whose  syllabus  is  shown  in  Table  4,  and 
we  also  recommend  that  the  material  used  for  lectures  is  translated  in  Kyrgyz,  the  official  language  of 
Kyrgyzstan. 


Table  4.  Plan  for  a training  course  for  faculty  members  in  BAFE 


Content 

Beginner 

Introduction 

Overview  on  the  use  of  ICT  technologies  for  educational  methods. 
Content  driven  learning,  blended  learning,  collaborative  learning. 
Introduction  to  Instructional  Design.  Examples. 

2h 

Moodle  practice  1 

Course  settings,  users  and  roles  management,  Moodle  web  page, 
file  and  directory  management. 

2h 

Moodle  practice  2 

Forum  (individual  and  group),  assignments. 

2h 

Moodle  practice  3 

Questionnaire  (questions  of  different  types,  building  a questionnaire). 

2h 

Moodle  practice  4 

Wiki  (individual  and  group),  glossary. 

2h 

Moodle  practice  5 

Course  design,  i.e.,  storyboard  of  the  course  with  a description 
of  the  modules,  planned  activities,  initial  forum  messages  for  the 
class,  activities  scheduling. 

2h 

Moodle  practice  6 

Learning  object  development:  select  one  of  the  module  defined  in 
the  design  phase,  develop  it,  and  upload  into  Moodle. 

2h 

Of  course,  the  success  of  the  use  of  an  LMS  cannot  be  measured  by  considering  only  its  adoption  by 
faculty  members,  because  it  is  the  community  of  students  that  plays  the  major  role.  In  the  case  of  BAFE  this 
means  organizing  also  brief  training  courses  on  the  use  of  Web  browsers,  to  facilitate  the  access  to  the 
on-line  platform.  Eirst  year  students  at  BAEE  are  offered  introductory  courses  on  Microsoft  Word  and 
PowerPoint  that  need  to  be  extended  with  lectures  on  how  to  access  to  the  Web,  especially  for  those  groups 
of  students  coming  from  rural  areas.  An  easier  access  to  the  network  is  of  course  an  essential  requirement  to 
allow  the  full  adoption  of  Moodle.  In  the  mean  time,  to  help  students  who  have  troubles  with  on-line 
assignments,  group  mates  could  help  their  peers  in  the  uploading  phase  on  scheduled  hours  in  computer  labs. 
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6.  CONCLUSION 

This  paper  has  presented  the  situation  at  UNIGE  where  a Moodle-based  LMS,  adopted  in  2004,  is  currently 
used  by  the  majority  of  faculty  members  and  students.  It  has  also  presented  the  current  situation  in  a different 
context,  where  initial  steps  have  been  taken  to  introduce  the  use  of  Moodle  for  teaching  and  learning,  with 
the  ultimate  goal  of  bridging  the  gap  between  education  and  innovation.  Taking  advantage  of  the  experience 
already  gained  in  UNIGE,  the  same  short  survey  has  been  administered  also  in  BAEE,  helping  identifying 
some  difficulties  but  also  some  expectations  for  faculty  members.  A training  course  for  educators  is  planned 
in  BAEE,  hoping  that  this  could  become  a driving  force  to  promote  changes  in  the  educational  practices.  Of 
course,  the  lack  of  a pervasive  infrastructure  and  the  high  cost  for  network  access  still  constitute  a barrier  to 
the  complete  adoption  of  LMS  in  education,  but  the  process  has  started  and  it  is  not  reversible.  Another  step 
in  BAEE  is  the  delivery  of  a survey  for  the  “other  side  of  the  coin”,  i.e.  the  community  of  students,  to 
understand  their  needs  and  desires.  The  results  of  this  survey  will  be  used  to  plan  a training  course  on  Moodle 
for  them  as  well.  Einally,  we  think  that  both  communities  (teachers  and  students)  should  also  become  aware 
of  the  potentiality  of  Web  2.0  and  collaborative  learning,  and  this  is  the  future  2.0  task. 
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ABSTRACT 

Our  aim  is  to  determine  which  teaching  methods  students  in  Computer  Science  and  Information  Systems  prefer.  There 
are  in  total  5 different  paradigms  (behaviorism,  cognitivism,  constructivism,  design-based  and  humanism)  with  32 
models  between  them.  Each  model  is  unique  and  states  different  learning  methods.  Recommendations  are  made  on 
methods  that  can  be  used  in  the  teaching  and  learning  of  Computer  Science  and  Information  Systems.  Learning  can  differ 
greatly  from  person  to  person.  Clearly  a teacher  cannot  possibly  meet  every  single  students  needs  when  it  comes  to 
optimal  learning.  The  best  and  maybe  the  only  possible  outcome  is  to  try  to  determine  what  methods  or  combination  of 
methods  a teacher  should  apply  to  suit  the  biggest  percentage  in  a class.  Students  have  fundamentally  changed  with 
regards  to  their  intellectual,  social,  motivational,  and  emotional  needs.  The  modern  student  not  only  uses  technology  on  a 
daily  basis,  but  has  become  dependent  on  it.  Elements  of  all  theories  need  to  be  incorporated  into  a paradigm  that  will  be 
sufficient  to  educate  the  Computer  Science  students  of  today.  Technology  must  be  used  in  the  most  optimal  way  to 
engage  with  students  and  to  keep  them  interested. 
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Computer  Science,  Education,  Learning  Paradigms. 


1.  INTRODUCTION  AND  PROBLEM  STATEMENT 

A study  done  at  the  University  of  the  Free  State,  South  Africa,  found  that  scholars  are  facing  extreme 
obstacles  in  the  academia  due  to  an  alarming  low  proficiency  in  literacy  skills  (Van  Staden,  2011).  Teachers 
usually  have  big  classes  and  each  student  has  their  own  psychological  profile.  A students’  psychological 
development  should  be  included  when  lessons  is  planned  (Cottrell,  2001;  Jain,  Tedman,  & Tedman,  2007; 
Laurillard,  1997).  In  combination  to  this,  the  students’  attitude  towards  learning  also  plays  a factor  in  the 
learning  process  (Cottrell,  2001;  Kong,  2008). 

The  proportion  of  school  leavers  attending  university  is  up  from  the  top  15%  to  40%  and  higher.  This 
means  that  the  range  of  students  in  a class  is  a lot  bigger,  ranging  from  the  smartest  to  more  challenging 
students.  In  South  Africa  the  student  population  has  increased  with  77%  from  the  472  000  in  1993  to  the  873 
000  in  2009  (Smit,  2011). 

Prensky  (2001)  explains  that  ‘digital  natives’  refers  to  a new  generation  of  students  who  have  since 
childhood  been  engulfed  in  technology.  The  term  ‘native’  indicates  that  these  students  ‘speak’  technology 
fluently  - the  digital  language  of  laptops,  hand-held  computers  and,  of  course  - the  internet.  Prensky 
concludes  that  modern  students,  with  such  a vastly  different  upbringing  to  a generation  only  ten  or  twenty 
years  older  than  themselves,  cannot  possibly  be  educated  in  the  same  way. 

In  South-Africa  88.9%  of  first  year  students  have  unrestricted  access  to  a mobile  phone  with  a camera, 
73.7%  have  access  to  a Bluetooth  modem  via  a mobile  phone,  61.1%  have  access  to  a desktop  computer  and 
a further  49.6%  have  access  to  a laptop  or  notebook  (Thinyana,  2010). 

Tapscott  (1998)  is  quoted  as  saying  in  his  book  that  “the  old  approach  (of  didactic  teaching)  is  ill-suited 
to  the  intellectual,  social,  motivational,  and  emotional  needs  of  the  new  generation”. 

The  norm  in  higher  level  educational  institutes  is  an  hour-long  lecture  (Beerman,  1996),  but  studies  have 
found  that  the  attention  span  and  learning  ability  of  students  decline  after  20  minutes  (Shroeder,  1993).  This 
results  in  students  only  recording  about  500  words  out  of  a lecture  consisting  of  5000  (Beerman,  1996). 
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New  environments  create  different  ways  of  using  learning  theories  in  order  to  achieve  maximum  learning 
ability.  New  generations  and  new  learning  theories  add  to  the  problem  (Jarvela  & Niemivirtab,  1999). 
Understanding  the  way  learning  takes  place  is  difficult,  and  this  is  complicated  by  changes  in  the  students’ 
environment  as  well  as  changing  learning  styles. 

The  use  of  e-learning  is  becoming  more  common,  but  lecturers  must  be  reminded  that  the  structure  of 
teaching  is  more  important  than  the  means  of  delivery,  and  that  e-learning  gives  ample  opportunity  to 
accommodate  the  different  cognitive  levels  and  preferences  of  learners  (Pange  & Pange,  2011).  Whether  we 
are  developing  programs  for  teaching  and  learning  where  e-learning  is  incorporated,  or  those  without 
e-learning,  knowledge  of  the  learning  paradigms  and  the  application  thereof  is  necessary. 

When  teaching  Computer  Science,  lecturers  are  faced  with  all  of  these  problems,  as  well  as  the 
difficulties  of  balancing  practice  and  theory  in  classes.  We  should  adapt  our  teaching  methods  to  possible 
changes.  There  are  a large  number  of  methods  that  can  be  used.  How  do  teachers  decide  which  methods  to 
use? 

Our  aim  is  to  determine  which  methods  students  in  Computer  Science  and  Information  Systems  prefer. 
Feedback  from  the  students  will  be  analyzed.  Recommendations  will  be  made  on  methods  that  can  be  used 
in  the  teaching  and  learning  of  Computer  Science  and  Information  Systems.  The  paper  starts  with  a problem 
statement  and  the  motivation  for  this  study.  Background  is  given  on  learning  paradigms  and  theories.  Then 
the  data  collection,  analysis  and  results  are  described,  followed  by  a conclusion. 


2.  LITERATURE  STUDY 

There  are  five  paradigms  that  address  the  different  approaches  to  learning;  these  are  behaviorism, 
cognitivism,  constructivism,  design-based  and  humanism.  Each  of  these  paradigms  has  theories  and  models. 
The  five  main  learning  paradigms  are:  Behaviorism,  Cognitivism,  Constructivism,  Design-Based  and 

Humanism. 

2.1  Behaviourism  Overview 

Behaviorism  consists  of  the  view  that  the  students  approach  towards  learning  is  formed  by  the  teachers’ 
treatment  of  the  student  (Guey,  Cheng,  & Shibata,  2010;  Quevedo-Terrero,  2009).  The  student  is  influenced 
by  positive,  negative  or  constructive  reinforcement.  All  concepts  can  be  broken  down  into  steps,  with  these 
steps  ranging  from  simple  to  difficult.  In  this  paradigm  it  is  essential  that  a teacher  ensures  that  a step  is  fully 
understood  by  the  students  before  moving  on  to  a more  complex  idea.  This  is  more  important  than  going  over 
and  over  a target  concept  (Guey,  Cheng,  & Shibata,  2010).  The  challenge  in  practice  will  be  to  take  both  the 
strong  and  weak  student  through  the  steps. 

Skinner  (1968),  viewed  as  the  father  of  radical  behaviorism,  said  that  teaching  as  a technology  functions 
through  the  arrangement  of  possibilities  of  reinforcement  under  which  behavior  changes.  Behaviorism,  thus, 
is  methodical  and  organized.  The  simple  fact  is  that,  as  a mere  reinforcing  mechanism,  the  teacher  is  out  of 
date.”  Ten  years  earlier  Skinner  (1958)  had  already  described  a ‘teaching  machine’  that  provided  a physical 
instantiation  of  what  behaviorism  is  all  about.  In  the  2U‘  century  students  come  standard  with  their  own 
instrumental  aid  in  the  form  of  laptops,  mobile  phones  and  tablet  PC’s. 

Behaviorism  could  make  the  most  of  this.  Behaviorism  is  advantageous  in  that  it  sets  objectives  that  are 
clear-cut.  Due  to  the  approach  being  so  specific,  success  is  mostly  easily  observable.  Lastly  it  ensures 
behavioral  practice  - not  just  theory,  and  works  best  for  helping  learners  to  attain  behavioral  skills.  The 
Behaviorist  theories  include  classical  conditioning,  GOMS  model,  operant  conditioning  and  Social  Learning 
theory.  Social  Learning  Theory  will  be  discussed  in  more  detail  later  on. 

2.2  Cognitivism  Overview 

Cognitivism  deals  in  the  ability  of  a person  to  logically  think  through  information  in  the  learning  process 
(Guey,  Cheng,  & Shibata,  2010;  Quevedo-Terrero,  2009).  In  contrast  with  behaviorism  (which  focuses  more 
on  knowledge  accumulation)  a cognitive  learning  process  laid  importance  on  the  discovery  process  for 
learning  and  more  importantly  understanding  to  occur. 
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Creativity  is  an  integral  part  of  this  paradigm  in  order  to  find  the  patterns  of  which  we  have  already 
spoken  (Guey,  Cheng,  & Shibata,  2010).  It  is  also  important  to  use  creative  structuring  of  the  material  in 
order  to  pull  the  student  through  the  topic  and  the  process  of  learning  by  the  discovery  process. 

Cognitvism  deals  with  the  cognitive  processes  involved  in  learning.  These  processes  include  induction, 
deduction,  rule  finding,  law  discovering  and  pattern  recognition. 

In  educating  students,  cognitivism  focuses  on  the  transmission  of  knowledge  of  the  objective  reality  of 
the  environment  from  the  lecturer  to  the  students.  At  the  end  of  the  day  students  should  have  the  same 
representation  of  reality  in  their  minds  as  the  lecturer.  Cognitivist  theories  include  assimilation  theory, 
attribution  theory,  cognitive  load  theory,  cognitive  theory  of  multimedia  learning,  component  display  theory, 
elaboration  Theory  and  Stage  Theory  of  Cognitive  development  - among  others. 

2.3  Constructivism  Overview 

Constructivism  is  when  new  information  is  processed  with  old  ideas  and  understanding  about  the  subject  and 
an  own  opinion  and  the  ability  to  draw  conclusions  is  developed  (Guey,  Cheng,  & Shibata,  2010). 
Constructivism  motivates  the  student  to  think  for  himself  (Quevedo-Terrero,  2009). 

Constructivism  can  be  described  as  the  capability  of  constructing  knowledge  rather  than  acquiring  it. 
Constructivism  as  a paradigm  or  worldview  posits  that  learning  is  an  active,  constructive  process. 

In  Computer  Science  it  was  found  that  a passive  student  with  no  participation  fails  in  learning.  A student 
needs  to  be  involved  in  the  thought  process  by  contributing  to  the  conversation  and  listening  to  other 
students.  Trial  and  error  is  the  best  way  for  a student  to  learn  skills  like  programming  (Quevedo-Terrero, 
2009;  Warin,  Kolski,  & Sagar,  201 1). 

According  to  Pange  and  Pange  (2011)  most  existing  e-learning  programs  follow  a constructivist 
approach. 

The  constructivist  pedagogy  involves  the  following  characteristics  (Richardson,  2003): 

• Attention  should  be  given  to  the  background  of  each  student. 

• Group  discussion  should  be  facilitated  to  explore  domain  elements  with  the  purpose  of  creating 
understanding  of  a topic. 

• Formal  domain  knowledge  can  be  introduced  at  specific  points.  Students  should  be  allowed  the 
opportunity  to  challenge  existing  beliefs.  Students’  meta-awareness  of  their  understandings  and 
learning  processes  should  be  developed. 

Constructivist  theories  include  Case-Based  Learning,  Cognitive  Apprenticeship,  Communities  of  practice. 
Discovery  Learning,  Goal  Based  Scenarios,  Social  Development  Theory  and  Situated  Learning. 

2.4  Design  - Based  Overview 

Design-Based  learning  theory’s  goal  is  to  find  a way  to  balance  the  theoretical  knowledge  and  the  way  it  is 
used  to  teach.  Design-Based  Research  is  a lens  or  set  of  analytical  techniques  that  balances  the  positivist  and 
interpretivist  paradigms  and  attempts  to  bridge  theory  and  practice  in  education. 

The  paradigm  design-based,  looks  at  the  relationship  between  the  empirical  study  a student  needs  and  the 
theoretical  knowledge  needed  to  implement  these  concepts  (Alvarez,  Alacron,  & Nussbaum,  2011;  Kong, 
2008).  It  involves  complex  programs  that  are  studied  in  educational  environment.  Combining  software 
designs  and  educational  studies  is  the  most  common  type  of  this  paradigm  (Kong,  2008). 

A common  feature  of  design-based  research  is  the  fact  that  the  purpose  is  to  produce  theories  on  the 
process  of  learning,  and  teaching.  The  second  feature  is  that  the  methodology  is  of  an  interventionist  nature. 
The  intent  is  to  investigate  ways  to  improve  education  by  developing  new  forms  of  learning  and  then 
studying  them.  Thirdly,  design  experiments  create  conditions  for  developing  theories  and  then  place  these 
theories  in  the  way  of  harm. 

Design-Based  strategies  include  the  ADDIE  Model  of  Instructional  Design  and  the  ARCS  Model  of 
Motivational  Design. 
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2.5  Humanism  Overview 

Every  person  studies  to  improve  their  knowledge  in  order  to  work  in  that  field  one  day.  Each  person  sees 
himself  as  having  a purpose  in  life.  Learning  will  further  us  in  this.  This  view,  the  humanism  paradigm,  is  not 
so  seriously  concerned  with  the  ability  to  learn  as  it  is  with  the  attitude  towards  learning  (Guey,  Cheng,  & 
Shibata,  2010). 

Humanism  is  a paradigm/philosophy/pedagogical  approach  that  believes  learning  is  viewed  as  a personal 
act  to  fulfill  one’s  potential.  “People  learn  more  easily  when  they  are  self-driven  and  have  a desire  to  acquire 
knowledge  to  improve  their  state  of  mind”  (DeCarvalho,  1991). 

The  objectives  of  the  humanistic  view  of  education  are  (Gage  & Berliner,  1991): 

• Promotion  of  positive  self-direction  and  independence 

• development  of  the  ability  to  take  responsibility  for  learned  knowledge 

• development  of  creativity,  curiosity  and  an  interest  in  the  arts 

• development  of  people  who  are  self-sufficient  and  self-actualized. 

Theories  included  in  Humanism  are  Experiential  Learning,  Eacilitative  Teaching  and  Maslow’s  Hierarchy 
of  Needs. 

2.6  Application  of  Paradigms  in  the  Teaching  and  Learning  of  Computer 
Science 

As  can  be  seen  from  the  above,  there  is  an  overwhelming  number  of  choices  available.  To  limit  these  for 
further  study,  fourth  year  Computer  Science  and  Information  System  students  at  the  North-West  University, 
South  Africa,  were  asked  to  study  and  compare  these  choices,  and  to  select  some  of  them  for  further  study. 
This  selection  was  made  using  the  knowledge  gained  during  the  study,  but  of  course  did  not  exclude  own 
experience  and  preferences.  Six  methods  were  selected  for  further  study. 

Looking  at  Behaviorism  and  the  Operant  condition  theory  in  particular,  a student  should  first  and 
foremost  be  handled  with  respect  and  a positive  attitude  toward  each  student.  Hard  work  is  often  repeated 
after  praise  is  received  for  an  accomplishment  (Quevedo-Terrero,  2009).  This  will  motivate  the  students  to 
work  harder  in  and  out  of  class  and  will  help  them  to  keep  focus  while  the  class  is  in  session.  Motivation  is 
often  a problem  with  e-learning,  and  learners  should  know  in  advance  the  goals  and  benefits  obtained  from 
the  process  and  the  relevance  thereof  (Pange  & Pange,  201 1). 

According  to  the  Cognitivism  paradigm,  and  Elaboration  theory,  the  entire  curriculum  should  be 
planned  in  such  a way  that  the  level  of  difficulty  increase  throughout  the  year,  starting  with  the  easiest 
concepts.  Elaboration  theory  approaches  education  by  first  taking  a ‘wide-angle’  view  of  the  study  material 
so  that  students  may  see  the  ‘big-picture’  or  holistic  view  of  what  must  be  mastered.  Each  segment  is  then 
elaborated  on  to  provide  a more  detailed  view. 

The  Assimilation  theory  of  the  Cognitivism  paradigm  firstly  states  that  it  is  better  for  the  learning 
process  if  a student  understand  the  work  rather  than  to  try  to  learn  it  merely  as  abstract  ideas.  When  planning 
a lesson,  special  attention  should  be  given  to  ensuring  that  the  contents  of  the  lesson  will  be  presented  in  the 
simplest  possible  manner  to  ensure  understanding.  A teacher  should  thus  plan  a lesson  by  identifying  a single 
idea  per  lesson  and  then  building  the  class  around  this,  structuring  each  (every)  new  idea  with  care.  The  first 
idea  should  be  thoroughly  described  and  a few  questions  should  be  asked  to  make  sure  the  students 
understand  the  core  idea  before  the  rest  of  the  information  follows.  The  rest  of  the  lesson  should  then  be  built 
on  this  core  idea.  During  all  this  a teacher  should  continuously  ask  questions  and  give  positive  feedback  if 
it’s  deserved.  Tests  should  be  discussed,  and  well  answered  questions  should  be  mentioned.  The  feedback 
should  be  handled  with  care  to  ensure  every  student  is  motivated  to  learn. 

Cognitive  Theory  of  Multimedia  Learning  is  the  method  of  learning  with  pictures  and  other  forms  of 
media. 

Problem-Based  Learning  on  the  other  hand  focuses  on  solving  real  world  problems  by  means  of 
teamwork  and  creative  thinking. 

Social  learning  theory  focuses  on  the  interaction  between  the  environment  and  the  student  (Taylor, 
2003).  Students  form  models  of  behavior  from  which  learning  ensues.  Learning  from  models  take  on 
different  forms,  including  new  behavior  patterns,  judgmental  standards,  cognitive  competencies  and 
generative  rules  for  creating  new  forms  of  behavior  (Bandura,  1989). 
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3.  DATA  COLLECTION  AND  ANALYSIS 

A questionnaire  was  answered  by  third  year  Computer  Science  students  at  the  Potchefstroom  campus  of  the 
North-West  University.  Fifty  one  out  of  a possible  eight  five  students  answered  the  questions. 

Both  open  questions  and  multi  choice  questions  were  posed  to  these  students.  The  open  questions  were 
grouped  into  classes  of  similar  answers.  The  multi  choice  questions  was  based  on  a predetermined  scale 
(1  -I  strongly  disagree,  2-1  disagree  somewhat,  3-1  have  no  strong  opinion  either  way,  4-1  agree 
somewhat,  5-1  strongly  agree  ). 

The  questionnaire  was  developed  after  a literature  study  on  the  paradigms  and  methods  was  done.  The 
questionnaire  covered  aspects  such  as  the  availability  of  technology,  current  methods  of  education  and 
receptiveness  towards  certain  aspects  of  learning  theories. 


4.  RESULTS  AND  RECOMMENDATIONS 

Results  can  be  discussed  under  the  headings  visual  aids  and  e-learning,  student  research,  group  work, 
sequencing  of  work,  motivation,  access  to  technology  and  general  comments. 

4.1  Visual  Aids  and  e-learning 

A majority  of  69%  of  the  learners  prefers  a lecture  that  uses  slides  and  the  other  31%  either  has  no  opinion  or 
prefers  no  slides  what  so  ever.  Results  further  showed  that  78%  of  the  students  preferred  visual  presentations 
instead  of  a verbal  lecture.  Only  7%  preferred  a verbal  lecture  and  15%  did  not  mind  which  method  was 
used  by  the  lecturer.  This  clearly  shows  that  students  in  Computer  Sciences  prefer  visual  aids  in  the 
classroom  above  a strictly  verbal  environment.  This  supports  the  Cognitive  Theory  of  Multimedia  Learning 
where  students  learn  more  efficiently  with  pictures  and  other  forms  of  media. 

One  aspect  of  education  that  has  stayed  decidedly  static  is  the  way  in  which  students  access  study 
material.  Written  notes,  textbooks  and,  in  some  cases,  PowerPoint  slides  made  available  to  students  are  all 
that  a student  has  to  come  home  to.  Once  the  lecture  is  over  the  student  loses  some  dimensions  of  the 
pedagogical  process,  namely  the  verbal  and  interactive  components  found  in  a classroom.  One  way  to  negate 
this  loss  of  dynamism  is  to  make  videos  or  sound  clips  available  to  students  online.  A factor  that  came  up 
even  though  it  was  not  directly  asked  is  that  lecturers  sometimes  do  not  speak  clearly  enough.  Two  students 
indicated  that  lecturers  speak  too  softly,  and  a further  10  students  complained  that  lecturers  sometimes  talk 
too  fast  in  the  classroom.  In  a diverse  country  such  as  South-Africa  this  problem  is  compounded  due  to  a 
wide  array  of  languages  being  spoken  at  home.  The  language  that  is  used  to  instruct  in  the  classroom  might 
be  only  the  second  or  third  language  for  many  students.  Therefore  e-learning  in  the  form  of  multimedia  tools 
placed  online  could  be  very  useful  in  taking  steps  to  engage  students  of  a new  generation. 

Two  questions  were  posed  to  the  sample  group  of  students.  The  first  was  aimed  at  measuring  how  open 
students  would  be  to  use  sound  bites  and  videos  to  assist  them  in  performing  tasks  or  completing 
assignments.  When  asked  whether  they  would  enjoy  the  opportunity  to  watch  instructional  videos  on 
platforms  such  as  YouTube  to  assist  in  completing  a task,  67%  of  students  agreed  while  28%  had  no  strong 
opinion  on  the  subject.  Only  4%  of  students  did  not  agree  that  this  would  be  a good  idea.  The  second 
question  tried  to  gauge  the  feeling  regarding  videos  and  sound  bites  recorded  in  class  being  made  available. 
When  asked  this  more  open  ended  question  whether  they  would  like  access  to  sound  bites  and  video  material 
from  classes  that  they  could  review  at  home,  all  but  one  student  said  that  it  would  be  very  helpful,  especially 
in  doing  revision.  The  reaction  to  this  proposed  use  of  technology  was  almost  overwhelming  with  students 
using  copious  amounts  of  exclamation  marks  and  adjectives  such  as  “great”,  “amazing”  and  “fantastic”. 

4.2  Student  Research 

Social  Learning  theory  advocates  self-regulation  and  the  internalization  of  motivating  factors.  One  way  to  get 
students  interested  in  the  work  being  taught  is  to  have  students  do  some  research  on  a subject.  Of  course  this 
idea  overlaps  somewhat  with  the  theory  of  Discovery  Learning  which  states  that  learning  is  an  information 
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processing  activity,  by  which  students  try  to  understand  their  environment.  Students  do  this  by  organizing 
and  categorizing  information  using  a coding  system.  The  most  effective  way  to  develop  a coding  system  is  to 
discover  it,  and  what  better  way  to  discover  information  than  to  do  some  research  and  then  summarize  it? 
Large  proportions,  namely  61%,  of  students  feel  that  they  would  memorize  a concept  more  effectively  if  they 
did  some  research  on  that  subject  themselves.  30%  of  students  had  no  strong  opinion  either  way  and,  only  a 
measly  9%  of  students  felt  that  they  would  not  learn  a concept  more  effectively  if  they  did  research  on  it. 

4.3  Group  Work 

Another  big  aspect  of  Social  Learning  theory  is  observing  the  behaviour  of  others  and  forming 
conceptualizations  from  these  observations.  The  easiest  way  to  engage  students  in  activities  that  enable  them 
to  learn  from  others  is  group  work.  Students  don’t  seem  to  have  a great  affinity  for  group  work  however,  with 
50%  of  students  indicating  that  they  would  not  like  to  do  practical  assignments  in  groups.  Extending  this 
question  to  whether  students  would  like  to  do  a presentation  to  the  rest  of  the  class  in  a group  increased  the 
negative  reaction  to  54%  of  all  students.  This  exposed  a very  interesting  pattern  and  one  that  is  greatly 
connected  to  social  learning  theory.  37%  of  students  felt  that  they  would  not  remember  material  effectively  if 
it  were  presented  to  the  class  by  another  student  while  35%  felt  they  would.  28%  had  no  opinion  either  way. 
In  contrast  to  this,  63%  of  students  felt  that  they  would  definitely  learn  something  from  seeing  another 
student  perform  a practical  task  in  class  - whether  that  task  is  performed  correctly  or  whether  it  is  performed 
incorrectly  and  then  corrected  by  the  lecturer.  This  anomaly  might  be  explained  by  the  next  two  statistics. 
Only  48%  of  students  indicated  that  they  ever  participate  in  classroom  discussions.  Only  54%  of  students 
indicated  that  they  would  volunteer  to  answer  a question  in  class,  even  if  they  knew  the  correct  answer.  So 
while  students  feel  that  they  would  be  able  to  learn  from  their  peers  in  an  interactive  classroom,  not  many 
students  are  very  keen  to  participate  themselves.  It  is  clear  that  there  are  definite  advantages  to  an  interactive 
classroom,  but  that  the  onus  is  on  the  lecturer  to  ensure  that  students  feel  safe  to  voice  their  opinions,  and  that 
all  students  are  given  the  platform  to  be  heard. 

4.4  Sequencing  of  Work 

Elaboration  theory  places  high  emphasis  on  first  presenting  a holistic  view  of  the  material  before  ‘zooming 
in’  on  specific  parts.  Students  tend  to  agree  with  59%  of  students  saying  they  would  like  to  know  the 
structure  of  the  whole  course  to  be  studied  beforehand.  89%  of  students  said  that  they  like  to  know  and  see 
during  lectures  how  the  current  material  fits  into  the  big  picture.  85%  of  students  indicated  that  they  would 
remember  concepts  better  when  they  are  first  explained  in  a more  general  fashion  and  then  in  more  detail. 
This  bodes  very  well  for  Elaboration  theory  as  a method  of  education  going  forward. 

While  Social  Learning  theory  might  be  more  suited  to  the  practical  side  of  Computer  Science,  elaboration 
theory  could  effectively  deal  with  learning  the  theory.  Even  more  practical  applications  might  be  taught  in 
this  manner.  Consider  the  theory  behind  object  orientated  programming.  Elaboration  theory  would  provide 
the  students  with  a holistic  view  of  how  problems  will  be  solved  using  the  object  oriented  approach  before 
elaborating  on  the  more  complex  details  of  how  to  program  a specific  component. 

4.5  Motivation 

Another  interesting  topic  that  arose  is  the  one  of  motivation.  Where  does  motivation  come  from?  The 
question  of  what  motivates  them  was  asked  directly  and  the  results  showed  that  60.87%  of  students  felt  that 
their  motivation  is  internalized.  The  factors  that  these  students  listed  as  motivation  include  personal  goals, 
attaining  good  marks,  securing  their  futures  or  financial  incentives  like  bursaries.  The  other  39.13%  of 
students  felt  that  their  motivation  comes  from  more  external  sources  like  their  families,  spiritual  motivation 
or  the  way  that  they  were  raised.  23.91%  of  these  students  specifically  cited  interesting  work  and  effective 
lecturing  techniques  as  motivation. 
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4.6  Access  to  Technology 

18.18%  of  students  owns  or  has  access  to  a personal  computer,  22.73%  of  students  own  or  have  access  to  a 
laptop  and  59.09%  of  students  have  access  to  both.  That  means  the  whole  sample  population  of  Computer 
Science  students  have  access  to  either  a laptop  or  a desktop  computer.  This  statistic  is  supported  by  the  fact 
that  100%  of  these  students  own  an  internet-capable  mobile  phone.  25%  of  the  students  own  a tablet  PC, 
97.73%  of  students  have  regular  access  to  the  internet  and  email.  Modern  students  live  and  learn  in  an  era 
where  technology  is  the  norm,  and  for  Computer  Science  students  this  is  compounded.  This  is  a positive, 
considering  the  possibility  of  using  e-learning  to  accommodate  different  students  with  different  learning 
styles. 

4.7  General  Comments 

Most  students  found  the  current  method  of  teaching  engaging  and  interesting,  citing  facts  such  as  the 
lecturers  being  experts  in  their  field,  lecturers  being  helpful,  the  students  having  a great  interest  in  the 
subjects  presented  and  lectures  being  well  organized  as  the  greatest  positives.  In  fact,  only  6 students  did  not 
answer  emphatically  in  the  positive,  with  only  two  students  saying  that  they  do  not  find  the  lectures  engaging 
at  all.  There  is  room  for  improvement  however.  Six  of  the  students  felt  that  technology  could  be  used  to  show 
practical  examples  and  applications  in  the  classroom.  Due  to  the  obvious  need  for  interaction  but  the 
unwillingness  of  students  to  stick  their  necks  out  in  a classroom  environment  there  is  a definite  need  for 
technological  assistance  in  making  lectures  more  interactive.  In  today’s  fast  paced,  internet  based  world 
where  information  is  available  at  the  click  of  a button,  students  are  looking  for  faster  and  more  effective 
feedback  from  lecturers.  They  want  prompt  replies  on  their  emails,  quick  feedback  on  tests  and  assignments 
and  immediate  answers  to  their  questions.  This  is  where  technology  could  streamline  the  education  process 
and  keep  students  interested,  and  where  e-learning  can  be  used  complimentary  to  contact  sessions. 

Some  of  the  main  points  that  can  be  concluded  from  this  study  are  presented  in  table  1,  where  the  second 
column  refers  to  the  relevant  paradigm  and  the  third  column  to  the  theory.  From  this  it  is  obvious  that 
different  paradigms  are  represented. 


Table  1.  Summary  of  main  points 


Teaching  and  learning 

1 Paradigm 

1 Theory 

Learners  prefers  visual  presentations  (for  example  slides,  videos,  sound 
clips). 

Learners  feel  that  they  would  memorize  a concept  more  effectively  if  they 
did  some  research  on  that  subject  themselves. 

Cognitivism 

Behaviourism 

Constructivism 

Cognitive  Theory  of 
Multimedia  Learning 
Social  Learning 
Theory  Discovery 
Learning 

Learners  feel  that  they  learn  from  seeing  another  student  perform  a practical 
task  in  class. 

Behaviourism 

Social  Learning 
Theory 

Learners  prefer  to  know  the  structure  of  the  whole  course  to  be  studied 
beforehand  and  to  see  how  the  current  material  fits  into  the  big  picture.  Ideas 
should  follow  logically  on  each  other  . 

Cognitivism 

Elaboration  theory 

It  is  important  that  teachers  create  a comfortable  environment  in  the  class. 

Humanism 

In  general 

5.  CONCLUSION 

It  is  important  that  in  a lesson,  ideas  should  follow  logically  on  each  other  and  that  a structure  of  ideas  should 
be  created  in  class.  Consider  giving  the  students  the  structure  in  assistance  with  their  preparations  for  tests 
and  exams.  The  choosing  of  a main  idea  and  keywords  to  build  on  this  is  paramount  in  planning  lessons  as  it 
was  found  that  most  of  students  use  these  in  their  learning  process. 

It  is  further  important  that  teachers  create  a comfortable  environment  for  students  in  the  class.  It  is  often 
found  that  students  won’t  answer  a question  for  two  reasons.  The  first  is  the  most  obvious,  they  don’t  know 
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the  answer.  The  second  was  found  during  the  processing  of  the  questionnaires.  Students  are  afraid  of 
appearing  stupid.  Teachers  can  make  the  class  more  comfortable  by  telling  a joke  or  two,  knowing  the 
students  names  or  steer  the  students  in  the  right  path  if  they  don’t  know  the  answer.  Lecturers  with  classes 
that  are  relatively  small  can  consider  giving  marks  for  participation  in  class  discussions.  This  will  not  only 
motivate  students  to  listen  in  class  but  may  inspire  them  to  prepare  for  lessons.  This  will  enable  them  to 
contribute  something  to  the  lesson.  Preparing  for  a lesson  will  also  help  them  to  identify  the  things  they  don’t 
understand  during  a lesson.  Once  this  happens  learning  will  be  increased. 

Elements  of  all  theories  need  to  be  incorporated  into  a paradigm  that  will  be  sufficient  to  educate  the 
Computer  Science  students  of  today.  Elements  of  one  theories,  for  example  Social  Learning  theory,  will  help 
to  engage,  interest  and  motivate  students  to  learn  the  course  material  that  has  been  sequenced  so  brilliantly 
according  to  ( for  example)  elaboration  theory  in  an  effective  way.  Aspects  of  Humanism  such  as  creating  the 
right  environment  are  vitally  important.  Elements  of  Constructivism  like  setting  goals,  getting  students  to 
discover  information  and  skills  and  Cognitive  Apprenticeship  not  only  overlaps  with  the  fundamentals  of 
Social  Learning  theory  but  supports  and  enhances  it  in  a big  way.  Other  theories  in  Cognitivism  also  run 
parallel  with  Elaboration  theory  such  as  creating  schemas  and  connecting  new  information  to  old  knowledge. 
No  one  theory  can  succeed  on  its  own. 

Students  have  changed  and  are  now  both  equipped  with  and  dependent  on  technology.  Computer  Science 
students  are  even  more  enraptured  with  the  latest  technological  tools  available  due  to  their  interest  in  the 
field.  Technology  must  be  used  in  the  most  optimal  way  to  engage  with  students  and  to  keep  them  interested. 
Using  technology  to  construct  a visually  attractive  lecture,  the  educator  has  ample  opportunity  to  demonstrate 
behavior  that  will  be  effective  in  the  field  of  Computer  Science,  both  in  the  practical  and  theoretical  arena.  If 
the  lecturer  demonstrated  the  correct  use  of  syntax  in  coding  an  algorithm  the  student  will  form  a model  of 
behavior  in  his  mind  containing  the  visual  and  verbal  instructions  and  processes  that  was  demonstrated  in  the 
classroom.  This  can  be  reinforced  with  interaction  by  questions  answered  or  a student  demonstrating  problem 
solving  in  the  class.  Also,  memory  is  reinforced  by  transforming  symbolic  representations  into  suitable 
actions-  that  is  practicing  the  study  material  at  home  through  homework  assignments.  Social  learning  theory 
places  a high  value  on  self-regulation,  so  lecturers  should  encourage  students  to  take  an  interest  in  the  field  of 
study  and  take  some  initiative  in  conducting  research  and  deepening  their  knowledge.  In  terms  of  group  work 
and  presentations  or  demonstrations  by  students  there  would  be  a solid  base  to  work  from  to  achieve 
education  through  altering  perceptions  of  the  students’  environment,  and  ultimately  changing  behavior. 

It  is  clear  from  research  that  learning  can  differ  greatly  from  person  to  person.  There  are  in  total  5 
different  paradigms  with  32  models  between  them.  Each  model  is  unique  and  states  a different  learning 
method.  Clearly  a teacher  cannot  possibly  meet  every  single  students  needs  when  it  comes  to  optimal 
learning.  The  best  and  maybe  the  only  possible  outcome  is  to  try  to  determine  what  methods  or  combination 
of  methods  a teacher  should  apply  to  suit  the  biggest  percentage  in  a class. 

Some  of  the  main  points  that  can  be  concluded  from  this  study  are  presented  in  table  1,  where  the  second 
column  refers  to  the  relevant  paradigm  and  the  third  column  to  the  theory.  Erom  this  it  is  obvious  that 
different  paradigms  are  represented. 

Elements  of  all  theories  need  to  be  incorporated  into  a paradigm  that  will  be  sufficient  to  educate  the 
Computer  Science  students  of  today.  No  one  theory  can  succeed  on  its  own.  Students  have  changed  and  are 
now  both  equipped  with  and  dependent  on  technology.  Technology  must  be  used  in  the  most  optimal  way  to 
engage  with  students  and  to  keep  them  interested.  With  e-learning  it  is  easy  to  include  structure  and  to 
present  an  overview  as  well  as  links  to  detail.  Links  to  videos,  sound  bites  and  graphics  can  easily  be 
included.  Different  learning  styles  can  be  accommodated.  Technology  could  streamline  the  education 
process  and  keep  students  interested,  and  e-learning  can  be  used  complimentary  to  contact  sessions. 
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ABSTRACT 

In  this  paper,  we  present  a way  of  teaching  AI  search  algorithms  in  a web-based  adaptive  educational  system.  Teaching  is 
based  on  interactive  examples  and  exercises.  Interactive  examples,  which  use  visualized  animations  to  present  AI  search 
algorithms  in  a step-by-step  way  with  explanations,  are  used  to  make  learning  more  attractive.  Practice  exercises,  which 
are  interactive  exercises  where  immediate  feedback  is  given  when  a student  makes  an  error,  but  further  help  is  optional, 
are  also  used.  So,  the  student  can  try  by  him/herself  to  correct  the  error  or  ask  for  help  from  the  system.  Finally,  the 
student  can  take  a test  consisting  of  assessment  exercises,  which  are  interactive,  but  no  help  is  provided.  The  result  of  the 
test  determines  the  student's  knowledge  level.  Evaluation  of  the  system  through  a pre-test/post-test  and 
experimental/control  group  method  gave  very  promising  results  about  learning  capabilities  of  the  method.  Also,  results  of 
a questionnaire  show  that  the  majority  of  the  students  liked  the  system  very  much. 


KEYWORDS 

Web-based  interactive  system,  interactive  e-leaming,  AI  search  algorithms,  adaptive  system 


1.  INTRODUCTION 

Numerous  e-learning  systems  have  been  developed  for  education  during  last  decades.  A requirement  of  e- 
learning  systems  is  their  ability  to  provide  personalization  that  is,  interact  with  learners  by  taking  into 
account  their  differences  in  knowledge  levels,  learning  styles,  preferences,  goals  and  so  on  (Wade  and 
Ashman  2007).  Intelligent  Tutoring  Systems  (ITSs)  constitute  a popular  type  of  intelligent  educational 
systems.  They  perform  their  tasks  mainly  based  on  Artificial  Intelligence  (AI)  methods.  Employed  AI 
methods  provide  the  representation  and  reasoning  basis  needed  to  accomplish  the  goals  of  ITSs.  More 
specifically,  AI  techniques  are  used  to  represent  theoretical  and  practical  knowledge  regarding  the  teaching 
subject.  Such  knowledge  provides  the  foundation  for  developing  mechanisms  for  customizing  the  learning 
process,  given  information  related  to  learners.  AI  techniques  also  represent  information  gathered  from 
learners’  interaction  with  the  system.  ITSs  are  able  to  intelligently  analyze  learner  responses  to  presented 
problems  and  questions.  They  are  also  able  to  intelligently  analyze  questions  posed  by  learners  to  the  system 
itself,  regarding  the  teaching  subject.  Such  intelligent  analysis  enables  provision  of  intelligent 
support/assistance/feedback  concerning  specific  parts  of  the  teaching  subject,  for  which  learner  knowledge  is 
assessed  to  be  incomplete  and/or  misconceived.  ITSs  have  been  used  with  great  success  in  many  challenging 
domains  to  offer  individualized  learning  to  the  students  and  have  demonstrated  remarkable  success  in  helping 
students  learn  challenging  content  and  strategies  (Woolf  2009).  Adaptive  Educational  Hypermedia  Systems 
(AEHSs)  are  also  systems  that  offer  personalized  education.  They  are  specifically  developed  for  hypertext 
environments  such  as  the  WWW  (Brusilovsky  et  al  1998).  Enhancing  AEHSs  with  aspects  and  techniques 
from  ITSs  creates  another  type  of  personalized  education  systems:  Adaptive  and  Intelligent  Educational 
Systems  (AIESs)  (Brusilovsky  and  Paylo  2003). 
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In  an  Artificial  Intelligence  (AI)  curriculum,  a fundamental  topic  is  "AI  search  algorithms".  It  is  vital  for 
students  to  get  a strong  understanding  of  the  way  the  algorithms  work  and  also  of  their  implementation  on 
various  search  problems.  However,  many  students  have  particular  difficulties  in  understanding  and 
implementing  AI  search  algorithms.  Students  often  have  difficulties  in  understanding  algorithms,  which  are 
explained  to  them  on  the  blackboard. 

In  this  paper,  we  present  how  an  adaptive  and  intelligent  web-based  system  assists  in  teaching/learning 
search  algorithms  in  the  context  of  an  AI  course.  The  system  provides  interactive  exercises  and  examples  to 
assist  students  to  learn  and  practice  by  graphically  implementing  blind  and  heuristic  search  algorithms.  The 
system  provides  a step-by-step  guidance  and  assistance  during  the  application  of  the  algorithms. 

The  rest  of  the  paper  is  organized  as  follows.  In  Section  2 related  work  is  presented.  Section  3 presents 
the  domain  knowledge  structure  and  the  user  interface  of  the  system.  Section  4 presents  the  teaching  method 
and  learning  materials.  Section  5 presents  an  evaluation  of  the  teaching  method  and  user  interface.  Finally, 
Section  6 concludes  the  paper  and  provides  a direction  for  future  work. 


2.  RELATED  WORK 

There  are  systems  developed  to  assist  teaching  and  learning  in  the  context  of  AI  curriculum.  In 
(Hatzilygeroudis  et  al  2005,  Hatzilygeroudis  et  al  2006)  a web-based  intelligent  educational  system  in  the 
context  of  an  AI  course  that  uses  AI  techniques  for  teaching  and  assessing  learners  is  presented. 
PATHFINDER  (Sanchez-Torrubia  et  al  2009)  is  a tool  for  actively  learning  Dijkstra’s  algorithm.  The 
highlighting  new  feature  provided  by  this  tool  is  an  animated  algorithm  visualization  panel.  It  shows,  on  the 
code,  the  current  step  the  student  is  executing  and  also  where  there  is  a user’s  mistake  within  the  algorithm 
running.  TRAKLA2  (Malmi  et  al  2004)  is  a system  for  automatically  assessing  visual  algorithm  simulation 
exercises.  TRAKLA2  provides  automatic  feedback  and  grading  and  allows  resubmission,  which  means  that 
students  can  correct  their  errors  in  real  time.  In  (Kordaki  et  al  2007),  an  educational  system,  called  Starting 
with  Algorithmic  Structures  (SAS),  designed  for  teaching  the  concept  of  algorithm  and  those  of  basic 
algorithmic  structures  to  beginners  is  presented.  Students  come  across  the  implementation  of  algorithms  in 
real  life  scenarios  and  the  system  offers  feedback  to  correct  their  answers.  Also,  in  (Naser  2008),  a 
visualization  tool  for  teaching  searching  algorithms  is  presented.  However,  none  of  the  above  works 


3.  ARTIFICIAL  INTELLIGENCE  TEACHING  SYSTEM 

We  developed  the  Artificial  Intelligence  Teaching  System  (AITS),  to  assist  learning  and  teaching  in  the 
context  of  the  “Artificial  Intelligence”  course  in  our  Department.  It  is  an  adaptive  and  intelligent  web-based 
educational  system  that  teaches  AI  aspects  and  uses  AI  techniques  for  personalized  learning  and  assessment 
of  the  students.  Among  others,  it  teaches  AI  search  algorithms.  We  present  in  this  paper  the  way  it  does  it  at 
an  educational  level. 


Search  Algorithms 

I Introduction  | 

Blind  Search  Algorithms 

I I BFS  algorithm 

I I DFS  algorithm 

Heuristic  Search 
Algorithms 


Hill  Climbing  algorithm  | 

B&B  algorithm  | 

A*  algorithm  | 

Figure  1 . Part  of  the  Tree  of  the  AI  Search  Algorithms  Domain. 
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3.1  Domain  Knowledge  Structure 

AITS  uses  a tree-like  structure  to  represent  the  domain  knowledge  related  to  the  AI  search  algorithms.  In 
Figure  1,  a part  of  the  tree  structure  of  the  AI  search  algorithms  domain  is  presented.  The  subject  itself  is  the 
root  of  the  tree.  The  subject  is  divided  in  sections  and  the  sections  into  subsections.  Each  sub-section  deals 
with  a number  of  concepts,  which  are  the  leaves  of  the  tree.  Subsections  may  have  some  concepts  in 
common.  The  domain  knowledge  tree,  described  above,  is  displayed,  as  far  as  the  subsections  level,  at  the 
navigation  area,  at  the  left  side  of  the  user  interface.  From  that  tree  the  student  can  choose  a learning  goal 
(subsection).  Each  subsection  corresponds  to  a learning  page.  Each  learning  page  deals  with  a number  of 
concepts.  So,  the  concept  pages  constitute  the  teaching  material  of  the  domain. 

3.2  User  Interface 


A student,  through  the  user  interface,  initially  subscribes  to  the  AITS.  After  subscription,  the  student  can,  at 
any  time,  enter  the  system  with  his/her  personal  data.  The  interface  of  AITS  is  illustrated  in  Figure  2.  It 
consists  of  two  main  areas:  navigation  area  and  content  area,  and  also  an  info  bar,  which  accommodates 
various  information.  This  information  concerns  the  username  with  which  the  student  is  connected  to  the 
system,  the  residence  time  of  the  system  and  the  current  date  and  time.  Also,  it  offers  to  the  student  facilities 
for  changing  personal  elements,  changing  the  learning  style,  view  statistics  and  general  information  about  the 
system,  annotation  capabilities  of  the  system,  and  logout  from  the  system.  Especially  for  the  statistics,  the 
student  has  the  opportunity  to  see  all  the  answers  he/she  gave  and  his/her  knowledge  level  for  each 
subsection,  subject,  and  concept. 


BliiKl  Search  Algorithms 


- • HEURISTIC  SEAROI 


Heuristic  Search  AlgofitiHns 


Breadth-First  Search -BfS 


.Content 
-i/  Area 


ID-Iterative  Deepening 


«Q 


V 


Navigation  Area 


Figure  2.  User  Interface  of  the  System 


4.  TEACHING  METHOD  AND  LEARNING  MATERIALS 

Teaching  is  based  on  the  traditional  theory-examples-exercises  paradigm  (although  the  students  can  follow 
their  own  way).  That  is,  for  each  topic,  the  theory  is  first  presented.  Then,  some  examples  are  given.  Finally, 
the  student  is  called  to  do  some  exercises.  Theory  consists  in  presenting  a number  of  concepts.  Those 
concepts  are  presented  in  a simple-to-complex  way.  That  is,  the  simple  concepts  are  presented  first  and  the 
complex  concepts  (that  require  the  knowledge  of  one  or  more  simpler  concepts)  are  presented  afterwards. 
This  is  depicted  in  the  ordered  list  presented  to  the  user,  which  however  is  randomly  accessible.  Furthermore, 
the  student  can  review  a previous  concept  at  any  time.  He/She  is  also  not  forced  to  follow  the  system’s  way 
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of  teaching,  but  can  make  his/her  own  choices  for  studying  a learning  page.  Any  time  a student  has  finished 
the  study  of  a particular  learning  page,  he/she  can  take  the  corresponding  concept-level  test. 

The  system  offers  theory,  examples  and  exercises  about  AI  search  algorithms.  More  specifically,  the 
system  offers  theory  about  blind  search  and  heuristic  search  algorithms  (breadth-first,  depth-first,  hill- 
climbing, branch-and-bound.  A*  etc),  interactive  examples  and  two  types  exercises:  practice  exercises  and 
assessment  exercises.  Practice  exercises  are  interactive  exercises  that  are  equipped  with  hints  and  guidance 
during  the  learning  sessions,  aiming  to  help  the  student  in  learning  AI  search  algorithms.  On  the  other  hand, 
assessment  exercises  are  interactive  exercises  that  are  used  to  examine  the  student's  progress  and 
comprehension.  For  example,  a test  on  an  AI  search  algorithm  concept  consists  of  a number  of  interactive 
assessment  exercises. 

After  a student  enters  the  system  can  study  the  theory  about  blind  search  and  heuristic  search  algorithms 
and  see  interactive  examples  on  the  selected  subjects.  The  student  can  select  to  interactively  deal  with  a 
practice  exercise  to  be  helped  in  learning  an  algorithm.  Finally,  he/she  can  take  a test  consisting  of 
assessment  exercises  to  see  what  his/her  knowledge  level  is. 


4.1  Interactive  Examples 


The  system  provides  interactive  examples  of  different  types  on  AI  search  algorithms.  Interactive  examples 
necessitate  the  step-by-step  application  of  an  algorithm.  Each  example  consists  of  a tree  or  a graph  with  an 
explanation  for  each  step  of  an  algorithm.  By  clinking  a button,  the  next  step  of  the  algorithm  and 
corresponding  details  of  the  current  step  are  shown.  The  examples  are  using  animation  to  present  the 
application  of  algorithms.  In  Figure  3,  an  interactive  example  on  depth-first  search  (dfs)  algorithm  is 
presented. 


Artificial  Intelligence  Teaching  System  (AITS) 


Date:  30.04.20i: 
Tfne:  0©  OlA  51 
Username:  test 


Personal  Information  Learning  Style  Statistics  Informations  Questionnaire  Contact  Logout 


Search  Frontier : (2, 3,4,5}  ^ <4  » ■ 

Closed  Set:{  1 } 


Figure  3.  An  interactive  example  on  depth-first  algorithm 

4.2  Practice  Exercises 

Practice  exercises  are  interactive  exercises  and  their  purpose  is  to  assist  students  in  understanding  the 
algorithms  through  implementing  AI  search  algorithms  in  an  interactive  way.  A practice  interactive  exercise 
presents  a graph  or  a tree  and  the  student  is  called  to  reach  a specific  goal  node  starting  from  its  root 
following  an  AI  search  algorithm.  The  student  selects  a node  by  clinking  on  it  and  the  system  provides 
guidance  and  feedback  at  student’s  actions  during  his/her  interaction.  The  student  can  request  for  system 
assistance  when  stuck,  by  clicking  on  a help  button.  Then,  the  system  provides  feedback  on  what  should  do 
next  and  also  cues  regarding  the  next  to  visit/select  node. 
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More  specifically,  as  soon  as  a student  selects  a node  in  a practice  exercise,  the  system  acts  as  follows: 

1.  If  the  node  choice  is  correct  (in  which  case  node  color  changes  to  green),  then  wait  for  the  student  to 
select  the  next  node  and  go  to  1 . 

2.  If  it  is  incorrect  (in  which  case  node  color  changes  to  red) 

2.1  Provide  proper  feedback,  i.e.  adapted  to  the  error  made. 

2.2  If  the  student  clicks  on  the  "show  solution"  button,  display  the  correct  algorithm  step. 

2.3  If  the  student  selects  a new  node,  go  to  1. 


Artificial  Intelligence  Teaching  System  (AITS) 


Personal  Information  Learning  Style  Statistics  Infonnations  Questionnaire  Contact  Logout 


Date;  30.04.201: 
Tine:  Ocb  OlA  51 
Username:  test 


AI  Algorithms 
• BLIND  SEARCH 


Blind  Search  Algorithms 


. ♦ HEURISTIC  SEARCH 


BFS 


Xv 


(A^KC-€KE-GXG4.) 

The  answer  is  notCoired 

Error  Type: 

State  Transition  Error 

Number  Error  The  number  of  transitions  is  wrong 
The  Iransiton  (C-B)  and  (0-0)  is  not  determined. 


Mwe  Infonnation  atxiut  Errors 


Figure  4.  A practice  exercise  breadth-first  algorithm 


Artificial  Intelligence  Teaching  System  (AITS) 


Dat^  30.04.20i: 
Time;  0<i>  12A  26 
Username:  test 


Change  Persorral  InformatiMt  Learning  Style  Statistics  Informations  Questionnaire  Contact  Logout 


AI  Algorithms 
• BUND  SEARCH 


Blind  Search  Algorithms 


• Depth-First  Search-DFS 

• Breadth-Frst  Search-BFS 

• ID-Iterative  Deepenng 

• HEURISTIC  SEARCH 


Heuristic  Seardi  Algorithms 


Branch  & Bound 

Apply  the  Branch  and  Bround  Algorithm 
Initial  State:  1 Fnal  State:  6 


States:  1-2-3-4 
The  on^wer  is  not  correct. 
The  sequence  (3-4)  is  wrong. 


Figure  5.  An  assessment  exercise  of  branch  & bound  algorithm 


The  practice  interactive  exercises  provide  immediate  feedback  after  a student’s  incorrect  actions,  because 
the  main  objective  is  to  assist  the  student  in  learning  about  search  algorithms.  Also,  AITS  monitors  student’s 
actions  and  behavior  and  keep  records  for  each  student,  regarding  the  number  of  hints  requested  for  each 
practice  exercise,  the  total  time  that  he/she  spent  to  answer  it  correctly  and  the  number  of  tries.  In  Figure  4,  a 
practice  exercise  on  breadth-first  search  (bfs)  algorithm  and  the  corresponding  feedback  and  error  report  is 
presented. 
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4.3  Assessment  Exercises 

Assessment  exercises  are  again  interactive  exercises  that  aim  to  assess  the  knowledge  level  of  students  in  the 
corresponding  concept.  Also,  it  does  not  provide  help  either  feedback  during  the  student's  interaction  with  the 
system.  An  assessment  interactive  exercise  presents  a graph  or  a tree  and  the  student  is  called  to  reach  a 
specific  goal  node  starting  from  a start  node  implementing  an  AI  search  algorithm.  The  student  selects  the 
nodes  (states)  by  clinking  on  them.  After  the  student  has  selected  the  sequence  of  the  nodes,  the  system 
provides  the  errors  made  by  the  student.  More  specifically,  AITS  presents  the  selection  of  the  nodes  that  is 
not  correct  and  provide  detailed  feedback  on  the  errors  made.  In  Figure  5,  an  assessment  exercise  is  depicted. 


5.  EVALUATION 

We  conducted  an  evaluation  study  of  the  system  during  the  AI  course  in  the  fall  semester  of  the  academic 
year  2011-2012.  The  students  were  attending  the  AI  course  and  had  attended  lectures  covering  AI  search 
algorithms.  The  participants  in  the  evaluation  study  were  40  undergraduate  students  from  those  enrolled  in 
the  course.  The  methodology  selected  to  evaluate  the  system  is  a pre-test/post-test,  experimental/control 
group  design,  where  the  control  group  used  a traditional  teaching  approach.  The  students  were  divided  into 
two  groups  of  20  students  each.  The  evaluation  process  was  as  follows: 

(1)  A pre-test  was  given  to  the  students  of  both  groups.  The  participants  answered  to  exercises  on  AI 
search  algorithms. 

(2)  Main  Experiment:  The  control  group  was  involved  in  the  traditional  learning  and  teaching  process 
about  AI  search  algorithms  in  the  AI  course.  The  students  solved  exercises  given  by  the  tutor 
individually  and  then  discussed  them  with  the  tutor.  The  participants  of  the  experimental  group  were 
given  access  to  AITS,  to  study  concepts  of  AI  search  algorithms  for  one  week.  The  system  provided 
exercises  and  feedback,  as  described  in  Section  4,  during  their  interaction  with  it.  Additionally,  the 
students  of  the  experimental  group  answered  to  a questionnaire  containing  questions  about  their 
experience  from  interacting  with  AITS. 

(3)  A post-test  was  given  to  the  students  of  both  groups.  All  the  participants  answered  again  to  exercises 
about  AI  search  algorithms. 


Table  1.  Descriptive  Statistics  of  Pre-Test  and  Post-Test 


Group 

N 

Mean 

SD 

SE 

Pre-Test 

Control 

20 

5.705 

1.448 

0.32 

Experimental 

20 

5.825 

1.37 

0.31 

Post-Test 

Control 

20 

6.3 

0.951 

0.21 

Experimental 

20 

7.65 

1.027 

0.23 

In  order  to  reliably  analyze  students’  performance,  an  independent  f-test  was  used  on  the  pre-test.  The 
mean  value  and  standard  deviation  of  the  pre-test  were  5.825  and  1.37  for  the  experimental  group  and  5.705 
and  1.448  for  the  control  group.  As  the  p-value  (significance  level)  was  0.584>0.05,  it  can  be  inferred  that  the 
two  groups  did  not  significantly  differ  prior  to  the  experiment.  That  is,  the  two  groups  of  students  had 
statistically  equivalent  performance  abilities  before  the  experiment.  In  Tables  1 and  2 the  descriptive 
statistics  are  presented  as  well  as  the  t-test  results  from  assessment  of  students’  learning  performance  (as 
calculated  by  the  SPSS  v.20  tool).  The  results  reveal  that  the  mean  value  of  the  pre-test  of  the  experimental 
group  is  higher  than  the  mean  value  of  the  pre-test  of  the  control  group.  The  Levene’s  test  confirmed  the 
equality  of  variances  of  the  control  and  experimental  groups  for  pre-test  (F=0.306,  p=0.584)  and  post-test 
(F=0.182,  p=0.672).  Also  the  t-test  result  (p=0.000  < .05)  shows  a significant  difference  between  the  two 
groups.  Thus,  it  implies  that  the  students  in  the  experimental  group  got  a deeper  understanding  of  the  AI 
search  algorithms. 


88 


lADIS  International  Conference  e-Learning  2013 


Table  2.  Descriptive  Statistics  of  Pre-Test  and  Post-Test 


Equality 


Levene's  Test 
for  Equality  of 
variances 

F Sig. 


t-Test  for  mean 


Df 


Sig.(2- 

tailed) 


MD 


95%  Confidence 
Interval  of  the 
Difference 


Lower 

Upper 

Pre-Test 

Equal 

0.306 

0.584 

-0.269 

38 

0.789 

-0.12 

-1.022 

0.7824 

Unequal 

-0.269 

37.8 

0.789 

-0.12 

-1.022 

0.7825 

Post-Test 

Equal 

0.182 

0.672 

-4.312 

38 

0.000 

-1.35 

-1.9838 

-0.7162 

Unequal 

-4.312 

37.78 

0.000 

-1.35 

-1.9839 

0.7161 

Moreover,  the  questionnaire  that  the  experimental  group  filled  in  included  9 questions.  Question  1 was  of 
multiple  choices  and  concerned  the  time  needed  for  a student  to  adapt  to  the  system.  Questions  2-5  were 
based  on  a Likert  scale  (1;  not  at  all,  5;  very  much)  (see  Table  3),  but  they  included  a ‘please  explain’  request 
too.  Finally,  Questions  6-9  were  of  open  type  and  concerned  strong  and  weak  points  or  problems  faced  in 
using  the  system. 

Their  answers  show  that  the  students  in  general  enjoyed  learning  with  the  system  (Q3,  Table  3).  Most  of 
them  (80%)  reported  that  they  needed  less  than  five  minutes  to  start  using  the  system  (Ql).  Also,  they  found 
that  the  user  interface  is  easy  to  use  (Q5,  Table  3). 

On  the  other  hand,  the  students  (75%)  agreed  that  the  system  helped  them  in  learning  about  AI  search 
algorithms.  Also,  due  to  this  fact  they  suggest  the  system  to  the  next  year  students  (80%)  (Q4,  Table  3). 


Table  3.  Questionnaire  Results 


Q 

Question 

Answers  (%)  I 

1 

2 

3 

4 

5 

2 

How  much  did  the  system  help  you  to 
learn  AI  search  algorithms? 

0 

5 

30 

40 

25 

3 

Did  you  enjoy  learning  with  the  system? 

0 

0 

20 

30 

40 

4 

Will  you  suggest  the  system  to  next  year 
students? 

0 

0 

20 

40 

40 

5 

Did  you  find  the  interface  easy  to  use? 

0 

0 

25 

40 

35 

In  general,  students  felt  that  the  system  was  helpful  and  easy  to  learn.  Moreover,  they  indicated  that  the 
system  helped  them  better  understand  the  AI  search  algorithms  and  learn,  especially  through  the  errors  they 
made. 


6.  CONCLUSION  AND  FUTURE  WORK 

In  this  paper,  we  present  an  adaptive  and  intelligent  web-based  educational  system,  called  AITS,  which  is 
used  for  teaching  AI  search  algorithms.  The  system  offers  theory  descriptions,  but  most  importantly 
interactive  examples  and  exercises.  A student  can  study  the  theory  and  the  interactive  examples,  which  use 
visualized  animation  to  present  AI  search  algorithms  in  a step-by-step  way,  to  make  them  more  attractive. 
Also,  some  explanations  are  given  during  interactive  examples  sessions.  Then,  the  student  can  try  practice 
exercises,  which  are  interactive  exercises  and  where  immediate  feedback  is  given  when  the  student  makes  an 
error,  but  further  help  is  optional.  So,  the  student  can  try  by  him/herself  to  correct  the  error  or  ask  for  help 
from  the  system.  At  a second  level,  the  system  provides  the  solution  to  each  step,  when  asked  by  the  student. 
Finally,  the  student  can  take  a test  consisting  of  assessment  exercises,  which  are  interactive,  but  no  help  is 
provided.  The  result  of  the  test  is  an  assessment  of  the  student's  knowledge  level  on  corresponding  concepts. 
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We  evaluated  the  system  through  a pre-test/post-test  and  experimental/control  group  method.  Results  are 
very  promising.  The  experimental  group  made  better  than  the  control  group.  Also,  results  of  a questionnaire 
given  to  the  experimental  group  show  that  the  majority  of  the  students  liked  the  system  very  much.  So,  it 
seems  that  visualized  animation  and  interactivity  are  two  crucial  factors  that  contribute  in  better  learning,  at 
least  for  subjects  like  AI  search  algorithms. 

A point  for  further  improvement  of  the  system  is  the  following.  At  the  moment,  the  system  offers  a rather 
limited  number  of  interactive  examples  and  exercises  of  both  types.  Implementation  of  such  learning 
materials  is  quite  time  consuming,  because  it  requires  programming.  So,  a way  for  semi-automatic  or 
automatic  generation  of  such  learning  elements  (actually  programs)  is  a quite  interesting  direction  for  further 
research.  This  is  one  of  our  next  research  efforts. 
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ABSTRACT 

The  digital  environment  has  had  an  immense  effect  on  American  society,  learning,  and  education:  we  have  more  sources 
available  at  our  fingertips  than  any  previous  generation.  Teaching  and  learning  with  these  new  sources,  however,  has 
been  a challenging  transition.  Students  are  confronted  with  an  ocean  of  digital  objects  and  need  skills  to  navigate  the 
World  Wide  Web  and  numerous  proprietary  databases.  Writing  and  disciplinary  habits  of  mind  are  more  important  than 
ever  in  this  environment,  so  how  do  we  teach  these  in  the  digital  age?  This  paper  examines  the  current  digital 
environment  that  humanities  faculty  face  in  their  teaching  and  explores  new  tools  that  might  support  collaborative  writing 
and  digital  skills  development  for  students.  In  particular,  this  paper  considers  the  effectiveness  of  a specially  configured 
multi-agent  wiki  system  for  writing  in  a large  lecture  humanities  course  and  explores  the  results  of  its  deployment  over 
two  years. 


KEYWORDS 
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1.  INTRODUCTION 

Students  in  entry-level  courses  at  large  public  universities  are  waiting  for  a new  model  of  collaborative 
learning.  They  hope  to  engage  with  the  rich  digital  media  all  around  them  and  to  participate  in  these  courses 
more  actively.  Students  expect,  but  often  cannot  find,  courses  that  build  digital  materials  around  their 
increasingly  collaborative  learning  styles.  We  developed  a demonstration  curriculum — enriched  and 
supported  through  our  multi-agent  technology — where  students  in  these  courses  work  exclusively  with  digital 
objects  and  produce  domain  specific  reflection,  collaborative  writing,  deep  engagement  with  the  subject,  and 
ongoing  self-assessment.  We  wanted  to  create  a new  classroom  for  large  lecture  courses  where  there  is  less 
anonymity  and  more  dynamic  learning,  where  students  truly  participate  in  a "community  of  scholars." 

The  goal  of  our  ClassroomWiki  experiment  was  two-fold.  First,  we  wanted  to  investigate  the  advantages 
and  disadvantages  of  using  ClassroomWiki  in  a large  classroom  for  implementing  collaborative  writing 
activities.  To  be  specific,  we  wanted  to  evaluate  the  ClassroomWiki’ s capability  of:  (1)  providing  accurate 
and  detailed  student  interaction  data  to  the  teacher  - leading  to  student  evaluations  that  better  reflect  the 
level/amount  of  contribution  towards  their  groups,  (2)  forming  student  groups  that  yield  higher  student 
participation  and  collaborative  learning  outcomes.  To  conduct  our  experiment,  we  divided  the  150  students  in 
a standard  U.S.  History  survey  course  into  two  sets:  control  and  treatment.  Both  control  and  treatment 
students  used  the  same  ClassroomWiki  interface  to  collaboratively  write  essays  on  a given  topic.  However, 
the  student  groups  in  the  control  set  were  formed  randomly,  and  the  treatment  set  student  groups  were  formed 
using  MHCF,  a multiagent  coalition  formation  algorithm. 

In  brief,  our  results  indicate  that  ClassroomWiki’ s treatment  set  students  interacted  more  with  the 
ClassroomWiki  site,  earned  slightly  higher  scores  than  the  control  set  students,  and  rated  their  peers  slightly 
better  compared  to  the  control  set  students.  Furthermore,  ClassroomWiki  was  able  to  track  more  detailed 
information  regarding  student  activities  and  provided  several  insights  into  the  collaboration  and  interaction 
patterns  of  the  participating  students.  In  addition,  the  student  scores  calculated  by  ClassroomWiki’ s detailed 
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student  contribution  summary  (e.g.,  the  summary  of  student  activities)  closely  represented  their  performances 
in  other  tests/assignments  in  the  class.  Finally,  in  the  responses  to  our  survey  and  self-assessment,  the 
students  slightly  favored  ClassroomWiki  over  Blackboard’s  wiki,  provided  several  constructive  comments 
that  will  help  us  improve  the  user-friendliness  and  usefulness  of  ClassroomWiki  further,  and  experienced 
affective  change  in  the  ways  they  think  about  the  teaching  and  learning  of  history. 


2.  LEARNING  ABOUT  DIGITAL  HISTORIES 

The  problem  in  most  large  lecture  courses  is  widely  recognized.  Students  are  assigned  little  writing  and  have 
few  opportunities  for  engagement  with  the  disciplinary  knowledge  they  will  need  to  be  successful.  Moreover, 
"course  management  software"  only  contributes  to  the  distance  that  students  feel  from  the  process  of  learning 
in  the  discipline  since  these  systems  offer  one-way  delivery  of  ever  increasing  amounts  and  kinds  of  content. 
There  is  a significant  gap  in  these  courses  between  content  and  assessment,  between  the  habits  of  mind  we 
hope  to  teach  and  the  activities  that  students  perform.  This  is  particularly  true  in  history,  where  large  numbers 
of  students  in  entry-level  surveys  are  taught  through  lectures,  textbook  assignments,  and  multiple  choice 
tests. 

Our  idea  was  to  create  a learning  system  that  spurs  collaborative  approaches— groups  of  students  will  be 
able  to  annotate  or  highlight  new  additions  to  digital  objects,  make  changes,  sort,  track,  and  compare 
versions,  as  well  as  rate  and  evaluate  all  aspects  of  their  group's  work.  Rather  than  write  analog  essays  that 
refer  to  digital  objects  (current  practice),  we  envisioned  students'  writing  digital  histories  in  a digitally  native 
system.  We  also  envisioned  collaboration  that  was  active,  meaningful,  and  sustained  through  the  practice  of 
joint  research  and  writing.  In  this  virtual  space  historical  knowledge  and  thinking  can  be  formed  through 
inquiry,  description,  analysis,  discussion,  argument,  and  collaboration. 

Our  vision  was  in  large  part  an  extension  of  our  longstanding  research  efforts  in  digital  history  and  more 
broadly  digital  humanities.  Ayers  (1999)  pointed  out  in  "The  pasts  and  futures  of  digital  history"  that  "as 
rapid  as  the  changes  have  been  . . . the  actual  writing  of  history  has  remained  virtually  untouched  and 
unchanged.  New  technology  has  not  affected  the  books  and  articles  that  form  the  foundation  of  what  we 
teach."  Much  of  this  statement  remains  true  today — historians  continue  to  write  history  in  books  and  articles 
in  the  same  way — but  the  foundation  of  what  historians  teach  has  shifted.  History  teachers  are  increasingly 
referring  to  materials  online,  weaving  them  into  their  assignments,  and  recognizing  the  importance  of 
primary  source  instruction  using  digital  sources.  Students,  of  course,  regularly  turn  first  to  online  searches  for 
historical  information. 

Ayers  (1999)  went  on  to  call  historians  to  a new  form  of  digital  writing:  "to  imagine  forms  of  narrative  on 
paper  that  convey  the  complexity  we  see  in  the  digital  archives."  Digital  history,  he  pointed  out,  "could  be 
both  a catalyst  and  a tool  in  the  creation  of  a more  literary  kind  of  history."  Ayers  was  not  alone  in  his 
suggestion  that  historians  write  digital  histories  for  the  digital  age.  Robert  Damton,  long  a leader  in  this  field 
and  a past  president  of  the  American  Historical  Association,  called  for  a "new  age  of  the  book"  in  1999.  He 
suggested  that  histories  might  "instead  of  using  an  argument  to  close  a case,  they  could  open  up  new  ways  of 
making  sense  of  the  evidence,  ...  a new  consciousness  of  the  complexities  involved  in  constructing  the  past." 
More  recently,  Ayers  (2013)  has  called  for  a "more  radical  online  revolution,"  in  which  undergraduates 
participate  in  and  produce  "generative  scholarship"  ("Toward  a More  Radical  Online  Revolution,"  The 
Chronicle  Review,  February  4,  2013). 

In  The  Journal  of  American  History  "Interchange:  The  Promise  of  Digital  History,"  Steven  Mintz  drew 
attention  to  the  ways  digital  technologies  have  reshaped  teaching:  "It  has  broadened  my  imagination.  I have 
embraced  audio  and  visual  sources  because  they  are  much  more  accessible  than  in  the  past.  I have  created 
interactive,  inquiry-  and  problem-based  teaching  and  learning  activities  on  the  Digital  History  Web  site 
because  new  technologies  encouraged  me  to  rethink  the  very  nature  of  history  teaching  and  to  reimagine  it  as 
active  engagement."  (Cohen  et  al.  2008)  Scholars  have  also  written  about  how  undergraduate  research  might 
spur  new  pedagogies  for  humanities  disciplines  in  the  digital  age  (e.g..  Crane  2012;  Pannapacker  2013; 
Thomas  2007,  Thomas  and  Ayers  2003;  Thomas  and  Liu  2012). 

Digital  histories  also  help  us  think  differently  about  how  we  know  what  we  know  about  the  past.  Digital 
historians  seeking  to  reconstruct  the  social  world  of,  say,  nineteenth-century  America  depend  on  these 
records,  but  by  importing  them  into  a digital  medium  these  historians  attempt  to  interrelate  and  shape  the 
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information  in  ways  that  might  make  invisible  histories  visible.  They  create  models  and  visualizations  about 
historical  questions,  and  attempt  to  uncover  patterns  and  relationships  not  otherwise  obvious.  If  we  are 
beginning  to  write  new  forms  of  history — digital  histories — should  we  expect  our  students  to  do  the  same,  to 
produce  their  analysis  in  these  potentially  exciting  new  formats?  (Cohen  and  Rosezweig  2006,  Seefeldt  and 
Thomas  2009,  Thomas  2007) 

The  ideas  behind  the  movement  in  Digital  History  have  coincided  with  changes  in  our  students' 
expectations  and  in  our  classrooms.  Marilyn  Lombardi  (2007a,  2007b)  has  recently  pointed  out  the 
difficulties  in  teaching  survey  courses — they  are  "juggernauts"  with  a "merciless  trajectory",  she  explains, 
and  depend  on  traditional  modes  of  assessment.  Research  studies  suggest  clearly  that  content  coverage  may 
be  less  valuable  than  independent  thinking  and  creativity  as  a means  toward  disciplinary  understanding.  But 
faculty,  she  also  points  out,  know  how  difficult  it  is  to  move  in  this  direction.  And  it  is  clear  that  a great  deal 
of  transformative  faculty  development  will  continue  to  be  necessary  to  spur  active  learning  pedagogy. 

Other  recent  studies  have  pointed  to  the  problem  of  digital  information  overload,  of  underdeveloped 
digital  literary  skills.  Palfrey  and  Gasser  (2010)  have  raised  awareness  about  the  ways  the  digital  generation 
consumes,  but  does  not  necessarily  produce,  digital  media.  Similarly,  in  the  field  of  history,  Martin  and 
Wineburg  (2008)  have  indicated  the  need  to  support  what  they  have  called  "the  novice  in  the  archive."  They 
ask,  "how  do  we  use  new  digital  technologies  not  only  to  make  sources  more  available,  but  also  to  cultivate 
skills  that  teach  students  to  read  and  think  about  these  sources  in  meaningful  ways?" 

At  the  University  of  Nebraska,  like  many  land  grant  institutions  in  the  U.S.,  the  survey  courses  in  U.S., 
European,  and  World  history  comprise  an  especially  high  demand  cluster.  Between  2001  and  2007  over 
25,680  students  took  these  courses,  averaging  just  over  3,800  students  per  year.  The  U.S.  History  courses 
held  the  majority  of  these  students;  indeed  over  2,500  students  on  average  took  the  U.S.  survey  courses  a 
year.  These  students  came  from  every  college  at  the  University — Art,  Architecture,  Business,  Engineering, 
Natural  Resources  and  Agriculture,  and  Arts  and  Sciences.  To  explore  and  evaluate  a new  model  for 
developing  historical  thinking  and  writing  in  the  large  lecture  U.S.  History  survey  course,  we  put  together  a 
collaborative  team.  The  team  included  faculty  in  Computer  Science  and  Engineering,  English,  and  History,  as 
well  as  graduate  students  in  History  and  Computer  Science  and  Engineering. 


3.  CLASSROOMWIKI 

ClassroomWiki  has  several  sets  of  features.  Eirst,  in  terms  of  communications,  it  supports  threaded  forums 
where  students  can  start  a discussion  thread  and  respond  to  discussion  threads.  Second  in  terms  of 
collaborative  interactions,  it  supports  versioning-based  collaborative  editing.  This  allows  contributing 
students  to  traverse  between  versions,  review  the  changes,  and  collaboratively  revise  their  Wiki  essays.  It 
also  has  a survey  instrument  for  peer  evaluation  of  their  group  activities.  To  further  support  students, 
ClassroomWiki  allows  for  searching  to  retrieve  archived  communication  (messages  and  discussion  threads) 
and  also  provides  automated  collaboration  reminders  to  notify  students  working  in  a group  of  any  changes  to 
their  group  pages. 

ClassroomWiki  has  a suite  of  instructional  support  features  such  as  a viewable  summary  of  student 
contributions,  teacher  announcements,  and  two  options  for  forming  groups:  random  and  intelligent.  The 
intelligent  group  formation  tool  uses  the  Multiagent  Human  Coalition  Eormation  (MHCE)  framework  (based 
on  the  principles  described  in  (Khandaker  and  Soh  2007)  to  form  heterogeneous  student  groups  using  the 
data  tracked  in  the  ClassroomWiki  environment.  The  novelty  of  this  MHCE  framework  is  in  its  group 
formation  process  which,  due  to  its  design  and  implementation,  1.)  adapts  to  the  changing  behavior  of  the 
students  and  2.)  balances  the  heterogeneity  of  the  members  so  that  a student  group  contains  students  of  all 
levels  of  performance.  That  is,  from  one  assignment  to  the  next,  MHCE  is  able  to  re-assign  students  into 
groups  for  the  next  assignment  based  on  how  they  behaved  or  collaborated  in  the  previous  assignment.  The 
behaviors  tracked  include  online  activities  such  as  the  number  of  contributions,  amount  of  words  contributed, 
quality  of  participation  in  threaded  discussions,  and  peer  evaluation  scores.  When  assigning  students  to 
groups,  MHCE  adopts  a heterogeneity  principle  (Khandaker  and  Soh  2010):  The  group  formation  algorithm 
should  balance  heterogeneity  of  learner  expertise  in  a group  in  such  a way  that  they  are  less  likely  to  give 
rise  to  situations  where  the  participating  learners  would  be  de-motivated  due  to  too  high  or  too  low 
incongruity  between  their  expertise  and  the  Wiki  artifacts  they  are  working  on. 

Eor  further  details  on  ClassroomWiki,  readers  are  referred  to  (Khandaker  et  al.  2010;  Khandaker  and  Soh 

2010). 
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4.  WRITING  ASSIGNMENTS 

In  Spring  2009  and  2010  the  lead  History  faculty  member  taught  a U.S.  survey  course  to  approximately  150 
students  and  introduced  group  writing  assignments  in  the  wiki  format — asking  the  groups  in  each  assignment 
to  use  library  accessed  databases  with  primary  source  documents  (Proquest  Historical  Newspapers)  to 
research  a topic,  and  to  collaborate  to  write  a thesis-driven  essay  explaining  its  historical  significance  based 
on  and  from  their  sources.  We  attempted  to  move  our  approach  to  digital  history  as  a research  enterprise 
directly  into  the  classroom.  The  specially  designed  wiki  system,  called  ClassroomWiki,  would  allow  us  a 
finer  granularity  of  analysis  and  with  it  we  could  test  the  effectiveness  of  the  group-building  algorithm  and  of 
the  step-by-step  instruction  in  historical  reading  and  analysis  or  "habits  of  mind."  Our  pedagogy  of  digital 
history  was  built  around  the  following  concepts:  1.)  emphasizing  writing  as  a process  of  learning  and 
disciplinary  thinking,  2.)  using  digital  primary  sources  to  expose  students  to  the  complexity  and  contingency 
of  the  past,  and  3.)  using  group  collaboration  to  formulate  interpretive  analysis  of  the  primary  sources. 

The  writing  assignments  were  carefully  sequenced.  We  began  by  helping  students  become  better  readers 
of  primary  documents  and  by  giving  them  writing  prompts  for  their  reading.  These  prompts  were  the  kinds  of 
questions  historians  ask  habitually:  who  wrote  this,  who  were  they  writing  to,  what  did  they  want  or  expect, 
what  keywords — i.e.,  important  words  or  subjects  in  the  text — were  used,  what  themes — i.e.,  broad  concepts 
that  the  author  or  creator  is  using  and  emphasizing  throughout  the  text  to  make  his/her  point — were  adopted, 
and  what  rhetorical  strategies — i.e.,  techniques  the  author  or  creator  is  using  to  convince,  persuade,  distort,  or 
arrange  evidence  in  his/her  favor — were  embedded  in  the  document?  To  help  students  understand  primary 
sources,  we  asked  them  to  evaluate  every  primary  source  in  four  steps: 

1 . Sourcing — find  out  who  wrote,  created,  or  produced  this  document,  when,  why  and  for  what  audience. 

2.  Contextualizing — situating  the  document  and  the  events  it  reports  in  place  and  time. 

3.  Corroborating — checking  other  documents  and  multiple  sources  to  assess  the  reliability  of  its  claims  and 
perspective. 

4.  Close  Reading — treading  carefully  to  consider  what  a source  says  and  the  language  used  to  say  it. 


5.  RESULTS 

For  each  group  writing  assignment  using  the  wikis,  we  built  in  required  steps  for  the  teamwork.  Students 
were  asked  to  evaluate  primary  sources  using  the  above  criteria,  conduct  research  using  digital  databases  to 
find  primary  sources  related  to  their  topic,  formulate  a thesis,  and  complete  a written  paper.  Students  worked 
with  entirely  digital  sources  and  could  mix  and  remix  them  into  their  analysis.  For  a given  assignment  hey 
conducted  their  initial  collaborative  work  in  ClassroomWiki's  "sandbox"  where  they  could  assemble 
materials  for  the  group  to  evaluate  and  share.  They  were  doing  history. 

Because  our  ClassroomWiki  system  had  the  ability  to  track  individual  level  work  at  a high  level  of 
granularity,  there  was  a correspondingly  higher  degree  of  individual  accountability.  Our  team's  main  interest 
was  in  the  viability  of  the  group  formation  agents  and  whether  the  algorithm  helped  groups  achieve  better 
performance.  We  found  that  the  multi-agent  system  did  result  in  slightly  but  statistically  significant 
improvement  in  a group's  success  when  compared  with  a control  group.  With  more  iteration  of  group 
activities  the  algorithm  would  probably  have  had  a greater  and  deeper  positive  effect. 

The  ClassroomWiki  system  also  provided  instructors  with  some  novel  and  useful  ways  to  see  the  work  as 
it  progressed.  The  system's  grading  and  commenting  functions  were  especially  useful  for  our  assessments. 
ClassroomWiki  tracked  the  total  word  counts  of  individual  students  (average  was  5,596),  the  numbers  of 
messages  posted,  and,  especially,  the  number  of  days  each  student  logged  onto  the  system  and  made  changes 
to  his  or  her  assignment  (average  6.4,  high  22  days,  low  2).  The  treatment  groups  earned  higher  average 
scores  and  a lower  standard  deviation  than  the  control  group,  suggesting  that  treatment  groups  were  better 
able  to  work  together  than  if  they  were  randomly  formed.  For  both  the  treatment  groups  and  the  control 
groups,  the  total  interaction  numbers  (counts)  were  remarkably  similar,  but  the  treatment  group  members 
rated  their  peers  (quality  of  participation)  slightly  better. 
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Table  1 shows  the  correlations  between  the  students’  scores  in  the  ClassroomWiki  assignment  and  their 
scores  in  the  other  tests/assignments  in  the  class.  The  values  in  Table  1 indicate  that  except  for  the  first 
document  analysis  assignment,  the  scores  the  students  received  in  the  ClassroomWiki  assignment  were  well 
correlated  with  their  scores  in  the  other  assignments/exams.  These  high  correlation  values  suggest  that 
individual  student  scores  that  were  calculated  based  on  ClassroomWiki’ s student  contribution  summary  (e.g., 
number  of  words  added/deleted,  number  of  forum  messages  posted,  etc.)  closely  represented  the  actual 
performance  of  the  students  in  the  other  tests  and  assignments  in  the  class.  Furthermore,  the  correlation 
values  in  Table  1 indicate  that  the  students’  final  exam  scores  were  higher  for  their  ClassroomWiki’ s score 
than  their  first  assignment  in  Blackboard's  Wiki  scores.  Since  the  final  exam  score  represents  the  knowledge 
and  understanding  of  the  students  that  they  gained  in  the  class,  the  higher  correlation  suggests  that  the 
detailed  tracking  of  student  behavior  in  ClassroomWiki  allowed  it  to  capture  the  performance  of  the  students 
more  closely  than  Blackboard’s  Wiki. 

Table  1.  Correlation  Values 


Test/Assignment 

Correlation  w/ 
ClassroomWiki  Scores 

Correlation  w/  Progressive  Era 
Group  Wiki  (Blackboard  Wiki) 
Scores 

Einal  05/01/09 

0.69 

0.54 

Midterm  Essay  Exam  03/02/09 

0.52 

0.67 

Civil  Rights  Essay  03/13/09 

0.51 

0.39 

Origins  of  Segregation  Document  Analysis  1/13/09 

0.30 

0.18 

Progressive  Era  Group  Wiki  (Blackboard  Wiki)  02/27/09 

0.29 

The  best  way  to  verify  ClassroomWiki’ s impact  on  the  performance  of  the  students  would  be  to  have  pre- 
test and  post-test  before  and  after  the  students  have  completed  ClassroomWiki’ s assignment  to  test  their 
knowledge  of  the  Wiki  topic.  However,  we  did  not  conduct  such  tests,  and  thus  it  is  difficult  to  verify  the 
impact  of  ClassroomWiki  on  the  students’  performance/learning.  As  a result,  we  have  compared  the  change 
in  students’  performances  before  and  after  the  first  Blackboard  Wiki  assignment  with  the  change  in  students’ 
performances  before  and  after  second  ClassroomWiki  assignment  to  estimate  the  impact  of  ClassroomWiki 
on  students’  performance  and  learning.  Table  2 shows  the  result  of  our  comparison. 

Table  2.  Change  in  Students’  Scores.  Note  that  the  median  for  BlackboardWiki’s  impact  was  0.00  because  there  was  no 
significant  improvement  in  the  control  set  students  from  the  viewpoint  of  the  median  value. 

Mean  Median 

BlackboardWiki’s  Impact:  Deltal  (Midterm  Score  - Document  Analysis  Score)  3.73  0.00 

ClassroomWiki’ s Impact:  Delta2  (Final  Exam  Score  - Midterm  Exam  Score)  2.71  3.00 


When  we  perform  a t-test  on  the  students’  scores,  we  see  that  the  average  change  in  students’ 
performances  due  to  Blackboard’s  Wiki  and  ClassroomWiki  were  not  statistically  significant.  However,  the 
median  of  the  change  of  the  students’  scores  improved  a little  after  they  completed  the  ClassroomWiki 
assignment.  This  slight  improvement  in  the  median  of  the  score  change  of  the  students  suggests  that 
ClassroomWiki  might  have  helped  some  of  the  low-performing  students  to  perform  at  a higher  level  and  thus 
improved  the  median  score.  Since,  our  results  are  not  conclusive,  we  need  to  collect  more  data  using  pre-  and 
post-  tests  in  our  next  experiment  to  better  understand  the  impact  of  ClassroomWiki  on  the  performance  and 
learning  of  students. 

We  also  categorized  the  students  according  to  the  change  in  their  scores  from  the  midterm  to  the  final 
exam.  Table  3 shows  our  results. 


Table  3.  Student  Scores  in  ClassroomWiki  Assignment.  Note  that  the  median  for  the  Control  Set  Students  was  0 because 
there  were  quite  few  control  set  students  who  did  not  complete/submit  assignments  and  received  a 0. 


Control  Set  Students 

Treatment  Set  Students 

All  Students 

Median 

Mean 

Median 

Mean 

Median 

Mean 

Students  with  No  change  in  the 
scores  (Einal  - Midterm) 

0.00 

31.17 

74.00 

59.42 

69.00 

50.00 

Students  with  Positive  Change 
(Pinal  - Midterm) 

85.00 

80.21 

83.00 

82.12 

85.00 

81.16 
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The  results  of  our  analysis  show  that  the  students  whose  performances  (i.e.,  scores)  improved  from  the 
midterm  to  the  final  (second  row  of  Table  3)  had  achieved  higher  mean  (difference  statistically  significant 
using  t-test  with  ^ 5%)  and  higher  median  scores  in  the  ClassroomWiki  assignment  than  the  students 

whose  performances:  (1)  did  not  change  or  (2)  deteriorated. 

In  self-assessment  and  self-evaluation  surveys  at  the  end  of  the  semester,  students  expressed  a high  level 
of  affective  change.  They  realized  significant  gains  in  historical  thinking,  analysis,  and  writing.  One  student 
wrote,  "It’s  one  thing  to  read  a history  text  book.  It’s  quite  another  to  play  the  part  of  historian  and  to  develop 
an  opinion  about  historical  events.  This  class  I felt  was  very  hands  on  in  that  enabled  me  to  be  a historian  for 
a little  while."  Another  explained,  "I  feel  that  the  document  analysis  assignments  and  the  wiki  assignments 
really  helped  advance  my  skills  in  writing  historical  analysis.  I have  never  been  in  a class  that  required  me  to 
write  historical  analyses  like  this  class  did."  Students  noted  that  group-writing  assignments  were  valuable 
because  they  expected  future  employers  to  emphasize  team  project  and  team  building.  "I  know  that  my 
ability  to  read  primary  documents  and  analyze  them  has  increased  immensely,"  one  student  explained,  "I  feel 
that  my  writing  quality  improved  throughout  the  semester  as  you  assigned  more  papers  regarding  analyzing 
primary  documents." 

But  students  also  expressed  considerable  frustration  with  learning  historical  habits  of  mind.  Students 
expected  a history  course  to  cover  events,  names,  and  dates.  They  expected  to  be  assessed  on  their  capacity  to 
recall  these  terms  and  perhaps  to  explain  their  significance.  As  a result  there  was  a considerable  gap  between 
what  students  expected  the  survey  course  to  be  and  what  we  were  asking  them  to  do.  One  student  expressed 
the  friction  between  these  expectations  in  this  way:  "First  of  all,  this  course  was  not  at  all  what  I expected.  I 
expected  a ‘standard’  history  class  where  the  information  is  presented,  notes  are  taken,  . . . While  at  first  I did 
not  like  the  format  of  the  class,  I actually  grew  to  like  it  and  really  appreciate  what  this  course  was  trying  to 
teach  me  in  terms  of  historical  facts  but  also  the  critical  skills  it  forced  me  to  acquire  and  utilize."  Another 
student  noted,  "In  the  beginning  I was  very  intimidated  by  the  document  analysis  projects  and  the  wikis.  I 
had  no  idea  how  to  analyze  historical  documents,  considering  things  like  perspective,  tone,  audience  etc.  . . " 
We  found,  therefore,  an  important  though  cautious  correlation.  When  students  were  divided  into  three 
groups — those  who  improved  from  the  mid-term  exam  to  the  final  exam  grade,  or  did  not,  or  stayed  the 
same — we  found  that  performance  on  the  second  wiki  correlated  with  individual  exam  performance  (using  t- 
test).  Significantly,  the  correlation  held  for  students  from  all  grade  ranges — that  is,  if  they  improved  from  C- 
to  C-H,  B to  A,  or  D to  C.  All  improvers,  in  other  words,  from  the  individually  assessed  mid  term  exam  to  the 
individually  assessed  final  exam  were  parts  of  higher  scoring  Wiki  groups  whose  make-up  had  been  tailored 
by  the  ClassroomWiki  algorithm,  and  the  relationship  was  not  accidental. 

The  distribution  and  size  of  the  bars  in  Figures  1 and  2 suggest  that  in  general,  the  students  were  able  to 
complete  their  assignment  in  ClassroomWiki.  Also,  a higher  number  of  students  in  the  control  group  did  not 
contribute  to  their  respective  groups  and  received  a 0.  This  could  be  an  indication  of  our  group  formation 
algorithm  improving  student  groups’  collaboration  by  bring  students  of  varying  competence  together  so  that 
no  group  is  formed  with  students  who  are  unable  to  collaborate  (due  to  lack  of  competence)  and  complete 
their  task.  This  claim  can  be  further  strengthened  by  comparing  the  scores  of  the  control  and  treatment  sets’ 
groups.  The  results  show  that  the  treatment  set  groups  earned  higher  average  scores  (treatment  = 74.67  vs. 
control  = 70.61)  and  had  lower  standard  deviation  (treatment  = 15.51  vs.  control  = 27.40,  p-value  <0.005). 
The  higher  average  and  lower  standard  deviation  suggest  that  the  treatment  set  student  groups  were  able  to 
work  more  effectively  together  as  a group. 


6.  SUMMARY 

Students  in  large  lecture  humanities  courses  need  to  gain  mastery  over  not  only  content  knowledge  but  also 
disciplinary  thinking.  They  should  in  these  courses  learn  to  master  a process — how  to  be  a historian,  a 
scientist,  a sociologist,  how  to  inhabit  a major  discipline.  Mastering  a process  requires  skill  development  and 
iteration,  and  mastery  once  learned  can  be  transferred  and  extended.  We  teach  introductory  survey  courses 
not  to  make  students  know  a set  of  facts,  but  to  teach  students  the  habits  of  mind  and  critical  inquiry  of  our 
discipline.  ClassroomWiki's  group  writing  algorithm  supported  the  development  of  disciplinary  thinking  in  a 
large  lecture  history  survey  course,  improved  tracking  of  student  work  in  groups,  and  may  have  led  to 
improved  performance  on  individual  graded  examinations  assessing  historical  thinking. 
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Figure  1.  Evaluation  score  distribution  of  ClassroomWiki  students 
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Figure  1.  Wiki  evaluation  score  of  students  in  ClassroomWiki 

Future  work  includes  conducting  more  studies  with  groups  formed  using  different  algorithms,  with 
writing  assignments  that  are  more  structured  and  spread  out  over  an  entire  semester,  and  with  additional 
features  such  as  tagging  and  rating.  We  are  also  interested  in  how  posting  previous  years’  Wiki  artifacts 
online  as  examples  impact  student  learning  as  well  as  their  performance  over  time. 


Avg.  Other  Evaluation  Score 
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ABSTRACT 

This  study  investigated  how  the  scientific  competencies  advocated  by  the  Programme  for  International  Student 
Assessment  (PISA)  could  be  improved  by  using  online  argumentation.  An  online  argumentation  system  served  as  an  aid 
for  argumentation  instruction  and  activities  among  students  during  the  experiment.  Seventy-one  10th  grade  high  school 
students  took  part  in  this  study.  A quasi-experimental  design  was  adopted  and  qualitative  and  quantitative  analyses  were 
used.  The  results  showed  that  using  the  online  argumentation  system  to  conduct  argumentation  instruction  and  activities 
could  improve  the  students'  PISA  scientific  competencies.  The  experimental  group  students  outperformed  their 
counterparts  in  terms  of  mean  scientific  competencies.  Finally,  this  study  proposed  suggestions  related  to  argumentation 
research  and  science  instruction. 


KEYWORDS 

Argumentation,  online  argumentation  system,  PISA,  scientific  competencies,  scientific  literacy 


1.  INTRODUCTION 

The  Programme  for  International  Student  Assessment  (PISA)  sponsored  by  the  Organization  for  Eeonomie 
Co-operation  and  Development  (OECD)  measures  students'  eompeteneies  in  reading  literaey,  mathematies 
literaey,  and  seienee  literaey  every  3 years.  Eaeh  survey  foeuses  on  a single  subjeet,  with  the  other  two  as 
supplementary  subjeets.  The  objeetive  of  the  PISA  is  to  understand  how  15-year-old  students  from  the 
partieipating  eountries  apply  knowledge  and  abilities  in  their  handling  of  daily  issues  before  eompleting  their 
eompulsory  edueation  (Bybee,  2008;  Bybee  & MaCrae,  2011;  Fensham,  2009;  Lin,  Hong,  & Huang,  2012a; 
Olsen  & Lie,  2011;  Sadler  & Zeidler,  2009).  Not  only  does  the  PISA  measure  the  students'  understanding  of 
sehool  eurrieula,  it  also  measures  the  important  knowledge  and  abilities  required  in  eivie  soeiety  for  their 
future  (Bybee,  2008;  Bybee  & MeCrae,  2011;  Lin  et  al.,  2012a;  Yeh  & She,  2010).  Nowadays,  publie  debates 
about  governmental  polieies  on  soeio-seientifie  issues  also  rely  on  the  seientifie  literaey  of  eitizens  to  make 
informed  deeisions  (Lin  et  al.,  2012a). 

The  PISA  foeuses  on  the  students'  eompeteneies  in  applying  their  aequired  knowledge  in  daily  seenarios 
(Bybee,  2008).  The  ability  to  probe  is  emphasized  in  their  seientifie  literaey,  ineluding  interest,  support,  and 
responsibility  (OECD,  2009).  Likewise,  Taiwan's  nine -year  integrated  eurrieulum  for  the  seienee  and 
teehnology  domain  is  similar  to  the  prineiples  of  the  PISA  in  terms  of  eultivating  the  required  eompeteneies 
in  students.  However,  the  global  rankings  of  Taiwanese  students  in  seientifie  literaey  fell  from  4th  plaee  in 
2006  (total  seore  of  534)  to  12th  in  2009  (total  seore  of  520)  (TWPISA  National  Center,  2012a).  The 
students'  performanee  seores  are  related  to  eomplex  faetors  sueh  as  teaehing  praetiee,  sehool  eulture,  and 
family  values.  Although  stating  that  the  international  evaluation  results  sueh  as  those  of  the  PISA  are  due  to 
edueation  performanees  is  eontroversial  (Wang  & Lin,  2005),  it  is  still  feasible  to  implement  relevant 
teaehing  strategies  in  the  elassroom  for  promoting  sueh  performanee.  The  enhaneement  of  PISA  seientifie 
eompeteneies  has  beeome  an  important  issue  in  the  reforming  of  sehool  eurrieula  by  Taiwan's  seienee 
edueators. 

Reeent  studies  on  the  PISA  have  foeused  on  the  effeets  of  demographie  variables  or  transnational  eulture 
(e.g.,  Basl,  2011;  Bybee  & MeCrae,  2011;  Kjamsli  & Lie,  2004;  Olsen,  2005;  Olsen  & Lie,  2011;  Nentwig, 
Roennebeek,  Sehoeps,  Rumann,  & Carstensen,  2009).  Few  studies  (e.g.,  Lavonen  & Laaksonen,  2009)  have 
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proposed  relevant  teaehing  strategies  for  improving  students'  PISA  seientifie  eompeteneies.  Researehers  need 
to  provide  more  experimental  data  in  terms  of  PISA  seientifie  eompeteneies  in  order  to  eontribute  to 
edueational  polieies.  Some  researehers  (e.g.,  Lee,  2009)  have  reeommended  the  improvement  of 
argumentation  skills  in  light  of  the  relationship  between  PISA  seientifie  eompeteneies  and  argumentation, 
whereby  it  is  thought  that  students  would  eultivate  their  seientifie  eompeteneies.  In  addition,  the  internet  is  an 
effeetive  tool  for  eondueting  argumentation  instruetion  and  aetivities  (Bell  & Linn  2000;  Clark  & Sampson, 
2008;  Lin,  Hong,  & Lawrenz,  2012b;  Yeh  & She,  2010)  and  new  generation  has  been  used  to  eontaet  in  the 
soeial  networking  serviee,  sueh  as  Faeebook.  The  present  study  eonsidered  the  use  of  internet  tools  and 
argumentation  as  a medium  to  explore  the  improvement  of  students'  seientifie  literaey. 

1.1  Scientific  Literacy 

The  OECD  (2009)  has  illustrated  the  definition  of  seientifie  literaey  and  its  framework  for  the  PISA  seienee 
assessment.  In  this  framework,  the  term  seientifie  literaey  denotes  an  overarehing  eompeteney  eomprising  a 
set  of  speeifie  seientifie  eompeteneies.  Seientifie  eompeteneies  inelude  the  abilities  to  mobilize  eognitive  and 
non-eognitive  resourees  in  any  given  eontext.  These  eompeteneies  are  regulated  by  the  individual's 
appreeiation,  interest,  values,  and  aetions  relative  to  seientifie  matters,  and  are  eharaeterized  as  erossing 
through  three  interrelated  aspeets,  ineluding  eontext,  knowledge,  and  attitudes  (as  shown  in  Fig.  1). 


Figure  1.  Framework  for  the  PISA  science  assessment  (OECD,  2009) 

Seientifie  eompeteneies  eomprise  a set  of  three  speeifie  construets  (OECD,  2009): 

1.  Identifying  seientifie  issues:  This  eonstmet  ineludes  reeognizing  issues  that  are  open  to  seientifie 
investigation;  identifying  keywords  to  use  in  searehing  for  seientifie  information;  and  reeognizing 
the  key  features  of  a seientifie  investigation. 

2.  Explaining  phenomena  seientifieally:  This  eonstmet  ineludes  applieation  of  seientifie  knowledge  in 
a given  situation;  deseribing  or  interpreting  phenomena  seientifieally  and  predieting  ehanges;  and 
identifying  appropriate  deseriptions,  explanations,  and  predietions. 

3.  Using  seientifie  evidenee:  This  eonstmet  ineludes  interpreting  seientifie  evidenee,  making  and 
eommunieating  eonelusions;  identifying  the  assumptions,  evidenee,  and  reasoning  behind 
eonelusions;  and  refleeting  on  the  soeietal  implieations  of  seienee  and  teehnologieal  developments. 

In  the  dimension  of  "eontext"  in  Fig.  I,  the  assessment  is  built  around  life  situations  ineluding  five 
aspeets:  health,  natural  resourees,  environment,  hazard,  and  frontiers  of  seienee  and  teehnology,  with  eaeh 
eontext  spanning  personal,  soeial,  and  global  seales.  The  dimension  of  "knowledge"  ineludes  "knowledge  of 
seienee"  and  "knowledge  about  seienee".  The  former  refers  to  knowledge  of  the  natural  world  while  the  latter 
entails  knowledge  of  seientifie  inquiry.  The  dimension  of  "attitudes"  ineludes  interest  in  seienee,  support  for 
seientifie  inquiry,  and  motivation  to  aet  responsibly  towards,  for  example,  natural  resourees  and 
environments. 
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1.2  Argumentation  and  Online  Argumentation 

Argumentation  in  scientific  learning  can  be  defined  as  a process  of  connecting  claims  and  data  through 
justification  or  through  the  evaluation  of  knowledge  claims  in  light  of  empirical  or  theoretical  evidence 
(Clark  & Sampson,  2009;  Jimenex-Aleixandre  & Erduran,  2008;  Osborne,  Erduran,  & Simon,  2004). 
Arguments  are  the  artifacts  of  argumentation  and  consist  of  either  assertions  or  conclusions,  including  their 
justifications,  reasons,  or  supporting  factors  (Zohar  & Nemet,  2002).  In  science  education,  argumentation  is 
seen  as  entailing  three  overlapping  goals:  making  sense  of  the  phenomenon  under  study  (i.e.,  constructing 
claims  and  explanations),  articulating  those  understandings  (presenting  arguments),  and  persuading  others  to 
adopt  one's  ideas  (critiquing  and  evaluating  counter  ideas  while  defending  one's  own)  (Borland  & Reiser, 
2009,  2011). 

Discourse  is  an  essential  component  of  argumentation,  and  social  interaction  plays  a critical  role  in 
knowledge  construction.  Such  social  interaction  forms  an  environment  for  discussion  which  allows 
individual  thinking  to  move  from  implicit  to  explicit,  and  this  can  result  in  group  reflection  to  reach  a 
common  consensus.  However,  students  are  often  not  able  to  propose  evidence  or  reasons  supporting  their 
arguments  (Nussbaum,  2002).  Argumentation  should  be  imparted  to  students  through  appropriate  teaching, 
learning  tasks,  assisting,  and  other  processes  (Kuhn,  1991).  It  would  achieve  such  teaching  objectives  more 
easily  by  complementing  them  with  scaffolding  (Bell  & Linn,  2000;  Nussbaum,  2002;  Osborne  et  ah,  2004). 

Argumentation  activities  require  specific  steps  and  thinking  processes,  of  which  and  information 
technology  can  assist  in  building  (Bell  & Linn,  2000;  Yeh  & She,  2010).  Computer  tools  can  assist  students 
in  becoming  researchers  and  reviewers,  helping  them  to  actively  evaluate  scientific  thinking  (Linn,  1998). 
Joiner  and  Jones  (2003)  believe  that  asynchronous  computer-mediated  communication  is  much  more 
advantageous  than  face-to-face  communication  and  is  suitable  to  become  the  cultivation  tool  of 
argumentation  skills.  For  example,  the  characteristic  of  asynchronous  online  discussion  is  the  extension  of 
answering  time  for  students  (Joiner  & Jones,  2003).  High  level  questions  can  stimulate  more  accurate  and 
definite  answers  from  students  when  given  a more  sufficient  waiting  time.  Moreover,  the  context  of 
argumentation  requires  equality  amongst  the  participants,  which  cannot  be  easily  established  in  traditional 
classrooms  (Duschl  & Osborne,  2002;  Joiner  & Jones,  2003).  Online  discussion  is  free  from  the  traditional 
classroom  constraints  whereby  the  discussion  is  led  by  a few  dominant  students,  and  low-achieving  students 
can  express  their  own  opinions  asynchronously.  Therefore,  this  study  hoped  to  complement  argumentation 
instruction  and  activities  with  the  advantages  of  an  online  environment,  and  observe  how  this  process  might 
promote  students'  scientific  literacy. 

1.3  Argumentation  and  Fostering  of  Scientific  Literacy 

Argumentation  is  the  coordination  of  evidence  and  theory  to  support  or  refute  an  explanatory  conclusion, 
model,  or  prediction  (Clark  & Sampson,  2009;  Osborne  et  ah,  2004).  Such  a process  requires  PISA  scientific 
competencies  in  using  scientific  evidence  and  explaining  phenomena  scientifically.  The  competency  in  using 
scientific  evidence  involves  accessing  scientific  information  and  producing  arguments  and  conclusions  based 
on  scientific  evidence.  The  competency  in  explaining  phenomena  scientifically  includes  describing, 
interpreting  phenomena,  and  predicting  changes,  and  may  involve  recognizing  or  identifying  appropriate 
descriptions,  explanations,  and  predictions  (OECD,  2009).  The  above  comparison  shows  the  potential 
relationship  between  argumentation  and  PISA  scientific  competencies.  Therefore,  conducting  argumentation 
instruction  and  activities  may  promote  the  students'  scientific  literacy. 

The  improvement  of  PISA  scientific  competencies  also  requires  students  to  go  through  the  scientific 
learning  process  in  the  scientific  context.  Previous  science  learning  studies  have  viewed  individual  learning 
as  a form  of  conceptual  change  (Posner,  Strike,  Hewson,  & Gertzog,  1982),  and  relevant  researchers  (Dole  & 
Sinatra,  1998;  Nussbaum  & Sinatra,  2003;  Yeh  & She,  2010)  believe  that  argumentation  can  effectively 
promote  students'  conceptual  change.  The  process  of  argumentation  allows  students  to  compare  and  contrast 
arguments  through  in  depth  thinking  (Dole  & Sinatra,  1998),  essentially  triggering  a cognitive  conflict  in  an 
individual  by  giving  him/her  new  information  that  varies  from  his/her  prior  knowledge.  This  is  the  first 
condition  proposed  by  Posner  et  al.  (1982)  for  conceptual  change:  the  learner  has  to  feel  dissatisfaction  on  a 
current  concept.  A cognitive  conflict  triggered  by  in-depth  thinking  can  then  lead  to  learning  and  conceptual 
change  (Nussbaum  & Sinatra,  2003). 
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1.4  Research  Question 

Based  on  the  above  literature  discussion,  this  research  used  an  online  argumentation  system  to  complement 
the  argumentation  instruction  and  discussion  of  PISA  issues.  This  study  investigated  the  improvement  of 
PISA  scientific  competencies  throughout  these  online  argumentation  processes.  The  research  question  was  as 
follows:  Were  there  any  differences  in  scientific  competencies  between  students  who  participated  in  online 
argumentation  instruction  and  activities  and  those  who  did  not? 


2.  METHODOLOGY 

2.1  Participants 

The  subjects  of  the  PISA  are  primarily  15-year-old  students  worldwide  who  have  completed  compulsory 
education.  Therefore,  this  study  selected  10th  grade  high  school  students  (15  years  old)  from  a senior  high 
school  situated  on  the  edge  of  the  Kaohsiung  city  center  in  southern  Taiwan.  This  study  was  conducted 
during  the  students'  first  summer  vacation  when  they  had  just  finished  their  junior  high  school  education.  A 
total  of  79  students  were  chosen  from  two  out  of  six  classes.  Due  to  eight  students  applying  for  leave  during 
the  experiment  process,  a total  of  71  students  participated  frilly  in  the  research  throughout,  with  36  of  them  in 
the  experimental  group  and  35  in  the  control  group.  These  students  had  similar  academic  achievements, 
having  being  promoted  to  senior  high  school  from  junior  high  through  the  national  entrance  examinations. 

2.2  Experimental  Design 

The  study  adopted  a quasi-experimental  design  (as  shown  in  Table  1).  A PISA  science  assessment  was 
conducted  on  both  groups  before  the  experiment.  The  students  in  the  experimental  group  then  went  through 
argumentation  instruction  and  activity  using  an  online  argumentation  system  for  a total  of  4 hours.  After  the 
experiment,  both  groups  conducted  the  same  PISA  science  assessment. 


Table  1.  Experimental  design  of  this  study 


Group 

Pre-test 

Treatment 

Post-test 

Experimental  group 

0, 

X 

O3 

Control  group 

O2 

O4 

Note:  Oi,  O2,  O3,  and  O4  were  the  PISA  science  assessments;  X was  the  online  argumentation 


2.3  The  Online  Argumentation  System 

The  argumentation  system  is  shown  in  Fig.  2.  The  upper  portion  shows  the  argumentation  topic,  for  which 
the  teacher  can  set  up  two  opposing  viewpoints.  The  lower  left  portion  is  the  column  for  the  tree  directory  of 
students'  arguments  titles.  This  directory  provides  a way  for  both  the  teacher  and  students  to  view  the 
arguments  of  all  the  students  in  the  class  by  means  of  hyperlinking  to  their  saved  location  in  a database.  In 
addition,  this  column  records  the  thought  process  of  all  the  students,  helping  individuals  to  reflect  on  their 
thoughts.  The  lower  right  portion  shows  the  argument  of  an  individual  student.  Students  are  able  to 
understand  the  perspectives  of  other  students  and  achieve  the  objective  of  mutual  interaction. 

As  shown  in  Fig.  3,  the  interface  for  students  to  construct  their  arguments  was  designed  based  on  the 
Toulmin  (1958)  Argument  Pattern  (TAP).  It  provides  the  scaffold  to  assist  students  in  formulating  their 
arguments.  To  ensure  that  students  are  not  affected  by  specific  terms  of  the  TAP,  Nussbaum  (2002) 
suggested  that  changes  to  spoken  language  should  be  made.  Students  are  given  partial  statements  with 
textboxes  next  to  them  where  they  can  type  information  to  complete  the  TAP  structure.  Separate  scaffolding 
provides  prompt  buttons  marked  with  a question  mark  for  the  students,  which  prompt  the  students  according 
to  the  missing  or  incorrect  components  of  their  previous  arguments.  If  a student's  information  did  not 
adequately  complete  the  statement,  the  button  remains  visible  and  indicates  that  the  information  was  not 
adequately  complete.  Should  the  preceding  argument  encompass  all  five  components,  then  the  buttons  would 
not  appear  to  achieve  the  "fading  out"  strategy. 
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Figure  2.  The  system  overview 


Figure  3.  The  scaffold  for  argumentation 


2.4  The  PISA  Science  Assessment 

The  released  items  of  the  PISA  science  assessment  can  be  downloaded  from  the  TWPISA  National  Center 
(TWPISA  National  Center,  2012b).  These  sample  items  were  constructed  or  translated  by  experts  and 
researchers  from  various  fields  in  the  past  when  Taiwan  participated  in  the  PISA.  The  most  items  are  the 
same  as  the  OECD  released  items,  which  can  be  downloaded  from  the  PISA  official  website  (OECD,  2006). 
Five  teachers  in  the  research  team  from  the  biological,  life  sciences,  physics,  chemistry,  and  earth  sciences 
field  were  tasked  to  select  items,  the  constructs  of  which  were  then  reviewed  and  verified  by  two  science 
educators.  To  achieve  constmct  validity,  the  Rasch  model  (Rasch,  1960)  was  applied  for  analysis  and  the 
infit  mean  square  error  (MNSQ)  of  each  item  was  calculated  to  be  between  0.95  and  1.18.  In  terms  of 
reliability,  the  item  separation  reliability  of  the  whole  test  was  0.95.  The  validity  and  reliability  were  both 
within  acceptable  range.  The  answering  formats  of  the  assessment  items  included  multiple  choice  questions, 
tme  or  false  questions,  and  short  answer  questions.  There  were  a total  of  15  items,  with  each  item  having  a 
full  score  of  10,  a partial  score  of  5,  and  a score  of  zero  for  a wrong  answer.  The  full  score  for  the  entire 
assessment  was  thus  150. 

2.5  The  Argumentation  Instruction  and  Activities 

Students  in  the  experimental  group  went  through  online  argumentation  instruction  and  activities. 
Argumentation  instruction  focused  on  how  to  use  the  argumentation  system  and  the  TAP  for  constructing 
individual  arguments.  During  the  instmction,  the  researcher  tried  to  allow  individual  students  to  construct  the 
arguments  that  best  suited  their  logical  thinking  by  following  the  teacher's  argument  example.  The  teaching 
strategy  enabled  students'  learning  to  progress  from  simplicity  to  complexity  and  from  single  to  diversity 
(Collins,  Brown,  & Newman,  1989)  in  terms  of  argument  construction. 

The  three  lessons  of  argumentation  instmction  were  conducted,  in  order,  using  a three  components 
argumentation  pattern  (claim-data-warrant),  a four  components  argumentation  pattern  (claim-data-warrant- 
backing),  and  a five  components  argumentation  pattern  (claim-data-warrant-backing-rebuttal).  The  templates 
used  for  student  practice  in  argumentation  instmction  were  based  on  examples  in  Toulmin's  book  (Toulmin, 
1958).  For  example,  "All  three  sisters  of  Ming  have  red  hairs  (warrant).  As  Hua  is  the  fourth  sister  of  Ming 
(data),  she  might  have  red  hair  as  well  (claim).  The  supporting  reason  is  that  their  family  members  have  red 
hair  genetically  (backing).  The  exception  is  that  Hua's  hair  turns  white  during  aging  (rebuttal)."  These 
examples  include  deductive  reasoning  and  inductive  reasoning.  The  researcher  explained  the  difference 
between  these  two  forms  of  reasoning  in  the  argumentation  instmction. 

Students  experienced  difficulties  mostly  in  responding  PISA  items  of  identifying  scientific  issues, 
especially  when  they  are  required  to  describe  specifically  about  the  treatment  and  dependent  variable  of  a 
contextual  investigation  (Lin  et  al.,  2012a).  The  final  lesson  of  current  study  conducted  argumentation 
activities  based  on  two  topics,  including  the  scientific  process  encompassing  experimental  design  and  control 
variables.  Both  topics  are  related  to  "identifying  scientific  issues",  to  compensate  PISA-related  competencies 
for  the  lack  of  training  in  this  area  for  argumentation  instmction.  During  the  argumentation,  the  teacher  did 
not  provide  any  correct  answer,  just  guiding  the  activities  instead. 
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3.  RESULTS 

The  tests  of  homogeneity  conducted  before  ANCOVA  obtained  the  following  results  for  “identifying 
scientific  issues”,  “explaining  phenomena  scientifically”,  “using  scientific  evidence”,  and  “overall  score”, 
respectively:  F(i^  6ir-02  (p^.870>.05),  67)^-15  (p=.695>.05),  F(i_  67)=-14  (p=.704>.05),  and  F(,^  67)=-30 

(/7=.583>.05).  These  results,  in  which  all  the  tests  did  not  reach  significance,  matched  the  basic  assumption  of 
regression  analyses.  Therefore,  ANCOVA  was  conducted  as  illustrated  below. 

The  ANCOVA  results  for  the  various  scientific  competencies  are  shown  in  Table  2.  Under  the  construct 
of  “identifying  scientific  issues”,  the  post-test  difference  between  groups  achieved  significance  68)^7.40, 
/7<.01)  with  an  effect  size  of  0.098.  According  to  Cohen’s  (1988)  definition,  the  small,  medium,  and  large 
effect  sizes  are  0.10,  0.25,  and  0.40,  respectively.  This  shows  that  the  experimental  treatment  had  a medium 
effect  on  the  competency  in  “identifying  scientific  issues”.  The  comparison  in  Table  2 shows  an  adjusted 
mean  post-test  score  of  26.85  for  the  experimental  group,  higher  than  the  21.81  obtained  for  the  control 
group. 


Table  2.  The  ANCOVA  of  scientific  competencies 


Competencies 

Group 

Pre-test 
Meant  SD) 

Post-test 
Meant  SD) 

Post-test 

Mean’tSE) 

F 

Cohen’ 

/ 

Identifying  scientific 
issues 

Experimental 

Control 

26.19(6.81) 

23.25(8.61) 

27.41(7.36) 

21.22(8.95) 

26.85(1.28) 

21.81(1.30) 

7.40** 

0.098 

0.330 

Explaining  phenomena 
scientifically 

Experimental 

Control 

30.88(9.14) 

28.17(8.32) 

33.83(7.38) 

30.62(8.22) 

33.18(1.10) 

31.29(1.12) 

1.47 

0.020 

0.143 

Using  scientific  evidence 

Experimental 

Control 

55.69(5.49) 

53.57(9.59) 

56.52(6.74) 

51.85(8.91) 

55.92(1.09) 

52.47(1.11) 

4.84* 

0.066 

0.266 

Overall  score 

Experimental 

Control 

112.50(12.96) 

105.00(20.51) 

117.92(14.01) 

103.14(18.91) 

116.04(2.40) 

105.06(2.44) 

1 — T 

10.00** 

0.128 

0.383 

Note:  *p<.05;  **p<.01;  5D=standard  deviation;  5£'=standard  error;  Cohen's/  =r|7(l-r|  ) 


Under  the  construct  of  “explaining  phenomena  scientifically”,  the  post-test  difference  between  groups  did 
not  achieve  significance  68)^1 -47,  p>.05)  with  a small  effect  size  of  0.143.  The  adjusted  mean  post-test 
score  of  the  experimental  group  was  33.18,  compared  to  31.29  for  the  control  group. 

Under  the  construct  of  “using  scientific  evidence”,  the  post-test  difference  between  groups  achieved 
significance  (F(i  68)^4.84,  p<.05)  with  an  medium  effect  size  of  0.266.  This  shows  that  the  experimental 
treatment  had  a medium  effect  on  the  competency  in  “using  scientific  evidence”.  The  comparison  in  Table  2 
shows  the  adjusted  mean  post-test  score  of  55.92  for  the  experimental  group,  which  is  higher  than  the  52.47 
obtained  for  the  control  group. 

Finally  under  “overall  score”,  the  post -test  difference  between  groups  achieved  significance 
68)=10.00,  p<.01)  with  an  effect  size  of  0.383.  This  showed  that  the  experimental  treatment  had  a medium  to 
large  effect  on  the  overall  scientific  competencies.  The  comparison  in  Table  2 shows  an  adjusted  mean  post- 
test score  of  116.04  for  the  experimental  group,  which  is  higher  than  the  105.06  obtained  for  the  control 
group. 


4.  DISCUSSIONS 

In  the  post-test,  the  mean  overall  scientific  competencies  of  the  experimental  group  was  higher  than  that  of 
control  group,  as  is  the  case  with  the  competencies  in  "using  scientific  evidence"  and  "identifying  scientific 
issues".  However,  the  experimental  group  students  did  not  outperform  their  counterparts  in  terms  of 
competency  in  "explaining  phenomena  scientifically".  The  nature  of  argumentation  is  to  produce  scientific 
evidence  to  support  conclusions,  and  the  process  of  argumentation  instruction  appeared  to  promote  the 
abilities  of  students  to  raise  scientific  evidence,  such  as  answering  questions  based  on  data  given  in  a 
diagram.  The  argumentation  topics  in  the  current  study  were  based  upon  identifying  scientific  issues,  and  the 
argumentation  process  appeared  to  promote  the  students'  abilities  to  identify  scientific  topics,  such  as 
understanding  control  variables  and  experimental  designs.  The  above  two  forms  of  competencies  do  not 
require  much  basic  background  knowledge  from  the  students  as  compared  to  the  competency  in  "explaining 
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phenomena  scientifieally".  Nevertheless,  some  items  of  the  "explaining  phenomena  scientifieally"  in  PISA 
require  students  to  have  basic  background  knowledge.  For  example,  there  is  a question  under  "Acid  Rain" 
that  states:  "Where  do  these  sulfur  oxides  and  nitrogen  oxides  in  the  air  come  from?"  The  official  answer 
from  PISA  is:  "Any  one  answer  referring  to  car  exhausts,  factory  emissions,  burning  fossil  fuels  such  as  oil 
and  coal,  gases  from  volcanoes,  or  other  similar  things."  The  scoring  of  such  questions  still  requires  students 
to  have  background  knowledge,  in  order  to  allow  students  to  answer  questions  effectively  by  applying  the 
competency  in  "explaining  phenomena  scientifically". 


5.  CONCLUSION 

In  summary,  argumentation  instruction  comprising  a specific  process,  online  scaffolding  assistance,  and 
argumentation  conflict  scenarios  may  help  to  promote  students'  PISA  scientific  competencies.  Therefore, 
using  the  online  environment  to  complement  argumentation  instruction  and  organizing  argumentation 
activities  focused  on  related  topics  is  a potential  direction  to  consider  in  the  future  for  improving  students' 
scientific  literacy. 
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ABSTRACT 

There  has  been  an  exponential  growth  of  e-learning  in  the  UK  Higher  Education.  However,  there  is  a growing  opinion 
that  e-learning  practitioners  are  not  critical  enough  and  that  they  should  adopt  a more  techno-negative  or 
techno-scepticism  approach  when  implementing  new  e-leaming  tools. 

In  this  paper  we  use  a techno-scepticism  framework  to  assess  the  evaluation  of  a newly  implemented  online  reading  list  at 
our  UK  Higher  Education  Institution.  In  particular,  we  look  at  the  ‘The  Myths  about  E-Learning  in  Higher  Education’ 
proposed  by  Njenga  and  Fourie  (2010). 

We  present  qualitative  and  quantitative  evaluation  data  collected  from  the  focus  groups  which  shows  that  the  students 
have  a very  positive  perception  of  this  newly  implemented  online  reading  list  and  believe  that  it  will  improve  their 
academic  experience.  Furthermore,  as  the  student  focus  groups  also  produced  rich  data  which  deal  with  the  students’ 
experience  of  e-learning  in  general,  five  of  the  myths  of  e-learning  that  relates  to  the  student  experience  are  used  to 
framework  these  discussions. 


KEYWORDS 

e-Learning,  Online  Reading  List;  Student  Experience;  Evaluation;  Techno-scepticism 


1.  INTRODUCTION 

There  has  been  an  exponential  growth  of  e-leaming  in  the  UK  Higher  Edueation  (JISC,  2004;  Bostoek,  2007; 
HEA,  2009;  HEFCE,  2009,  UCISA,  2012)  and  Worldwide  (Edueause,  2012).  For  example,  the  Virtual 
Learning  Environment  (VLB)  has  beeome  an  integral  part  of  the  student  experienee  of  Higher  Edueation 
(HRC,  2009;  NUS,  2011;  Quinsee  & Bullimore,  2011,  UCISA,  2012).  This  is  not  by  aeeident;  aeross  the 
seetor  e-Leaming  has  had  a signifieant  impaet  on  Higher  Edueation’ s eeonomieal  and  soeial  dimensions  and 
most  importantly  the  pedagogieal  dimension  (Chang  & Tung,  2008;  Hartley  et  al,  2011;  Barber  et  al,  2013; 
Stubbs,  2013).  As  sueh  there  exist  a number  of  organizations  both  nationally  (Assoeiation  for  Learning 
Teehnology  and  Jise);  and  internationally  (International  Assoeiation  for  Development  of  the  Information 
Soeiety  lADIS)  that  promote  teehnology  in  edueation. 

Our  higher  edueation  institution  is  eurrently  going  through  signifieant  ehange.  At  the  institution,  e- 
leaming  is  playing  a bigger  role  in  improving  the  student  experienee  (Quinsee  & Bullimore,  2011).  Parallel 
to  this  eontinuous  investment  in  learning  teehnologies,  the  Library  is  eurrently  implementing  a new  ‘online 
reading  list’. 

It  is  reeognised  by  edueationalists  that  reading  lists  are  an  integral  part  of  a student’s  higher  edueation 
aeademie  experienee  (Stokes  & Martin,  2008;  Bartlett,  2010;  Piseioneri  & Hlavae,  2012).  Reading  lists  are 
study  tools  eomprised  of  lists  of  books,  artieles,  journals,  websites  and  other  resourees  that  students  should 
read  and  refer  to  as  part  of  their  eourse,  and  are  generally  split  into  seetions  based  on  either  their  importanee 
to  the  eourse,  by  subjeet,  by  time  period  (e.g.  weekly  readings),  or  a eombination  of  all  three. 
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Reading  lists  are  therefore  used  by  aeademies  to  eommunieate  vital  information  about  reading  for  their 
modules  to  their  students  and  to  other  stakeholders;  for  instanee,  at  our  University,  reading  list’s  key 
stakeholders  are  students,  aeademies,  sehool  administrative  staff  and  library  staff,  partieularly  Subjeet 
Librarians,  who  are  responsible  for  liaising  with  and  building  eolleetions  for  speeifie  sehools  and 
departments. 

However,  students  are  the  primary  stakeholders  beeause  reading  list  plays  an  important  role  in  their 
aeademie  experienee  at  higher  edueation.  Considering  the  rapid  ehange  in  the  UK  higher  edueation  seetor, 
the  student  experienee  has  never  been  more  important.  As  sueh,  any  teehnology  that  attempts  to  faeilitate  this 
aeademie  experienee  should  be  eonsidered  as  an  e-leaming  tool.  This  understanding  is  in  line  with  UCISA’s 
(2012)  definition  of  Teehnology  Enabled  Learning: 

“Any  online  faeility  or  system  that  direetly  supports  learning  and  teaehing”  (page  2) 

Our  new  online  reading  list  (branded  Reading  List  Online  or  RLO  in  short),  powered  by  Talis  Aspire, 
allows  aeademies  and  the  library  to  manage  aeademie  reading  lists  online.  The  fundamental  aspeet  of  the  tool 
is  that  it  is  online;  meaning  that  students  ean  aeeess  their  module/  eourse  reading  lists  anytime  from 
anywhere.  This  eloud  storage  also  means  that  Talis  Aspire  ean  ereate  a eommunity  of  reading  lists  aeross 
different  institution  so  that  students  have  aeeess  to  reading  lists  beyond  their  partieular  institution.  It  is 
important  to  note  that  although,  this  tool  has  the  potential  to  be  ‘soeiaT,  it  is  not  a soeial  media  tool  at 
present. 

Although  there  is  an  inereasing  number  of  UK  universities  that  have  adopted  an  online  reading  list  with 
the  purpose  of  improve  the  student  experienee  (Talis  Aspire,  2013),  there  is  a growing  opinion  that  e-leaming 
praetitioners  are  not  eritieal  enough  when  implementing  new  e-leaming  teehnologies.  It  is  suggested  that  we 
should  adopt  a more  ‘teehno-negative’  or  ‘teehno-seeptieism’  approaeh  when  implementing  new  e-leaming 
tools.  In  partieular,  Njenga  and  Fourie  (2010)  argue  that  we  should  have  a more  eritieal  and  informative 
diseussion  on  e-leaming  adoption  in  higher  edueation. 

As  sueh,  the  foeus  of  this  paper  is  a ease  study  of  the  implementation  and  evaluation  of  a new  online 
reading  list.  The  data  is  used  to  first  present  the  students  pereeptions  and  aeeeptanee  of  the  teehnology  and 
then  textual  analysis  is  applied  to  assesses  five  of  ‘The  Myths  about  E-Leaming  in  Higher  Edueation’ 
proposed  by  Njenga  and  Fourie  (2010). 


2.  BODY  OF  PAPER 
2.1  Myths  of  e-learning 

Njenga  and  Fourie’s  (2010)  opinion  is  also  voieed  by  Kinehin  (2012)  who  ealls  for  ‘avoiding  teehnology- 
enhaneed  non-learning’;  pointing  out  that  although  the  students  aeademie  experienee  should  be  the  primary 
motivation  for  the  implementation  of  a new  e-leaming  tool,  this  is  not  always  the  ease.  e-Leaming 
praetitioners  are  themselves  aware  of  this  dilemma  (Psotka,  2013),  with  Comrie  (2013)  reeently  asking:  “So, 
how  do  we  ensure  learning  exploits  teehnology  and  not  viee  versa?” 

In  partieular,  Njenga  and  Fourie  (2010)  propose  that  there  are  ten  prevailing  myths  about  e-leaming  in 
higher  edueation.  A myth  is  defined  as  ‘an  idea  or  story  that  is  believed  by  many  people  but  that  is  not  tme” 
(Merriam- Webster,  2013).  These  myths  are: 

1.  e-Leaming  is  a savour;  its  redemptive  power  is  overreaehing  and  every  edueational  institution 
should  adopt  it 

2.  e-Leaming  ean  replaee  human  interaetion 

3.  e-Leaming  euts  the  eost  of  edueation,  for  instanee,  e-leaming  eourses  are  eheaper  to  deliver  than 
the  traditional  faee-to-faee  or  distanee  learning 

4.  Providing  numerous  eourses  and  an  abundanee  of  information  is  benefieial  and  ean  enhanee 
learning 

5.  ICTs  should  beeome  the  primary  medium  of  learning  in  higher  edueation 

6.  Leisure  (ineluding  playing  and  entertaining)  and  learning  are  separate  aetivities 
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7.  e-Leaming  will  make  HEIs  more  competitive  and  they  must  seize  it  or  be  declared  institutionally 
redundant 

8.  Establishing  the  infrastructure  (hardware  and  software)  in  e-leaming  is  the  most  difficult  part 

9.  e-Leaming  will  see  the  demise  of  traditional  campuses 

10.  e-Leaming  can  decrease  absenteeism  and  lower  dropout  rates  among  students 

According  to  the  authors,  these  myths  are  promoted  by  technopositivist  but  are  not  true  to  reality.  They 
define  ‘technopositivists’  as  those  people  that  repeatedly  create  and  propagate  a ‘compulsory  enthusiasm’ 
about  e-leaming  in  higher  education  for  personal  motives  and  without  exploring  the  dangers  and  rewards  of 
an  e-leaming  tool  for  teaching  and  learning. 

The  focus  of  our  evaluation  was  the  student  experience  of  the  new  online  reading  list.  More  specifically, 
the  aims  of  the  evaluation  were  to: 

• Discover  if  RLO  is  an  useful  and  effective  education  tool  for  our  students; 

• Measure  and  predict  students  acceptance  of  RLO 

• Understand  how  our  students  interact  with  their  traditional  paper  based  reading  lists  and 
significantly  how  this  academic  experience  can  be  improved  with  RLO; 

• Find  out  how  our  students  interact  with  the  library  in  order  to  obtain  items  from  their  reading  lists 
and  how  the  Library  Services  can  improve  this  process; 

• Gain  access  to  insight  that  could  be  translated  to  tangible  communication  strategies  to  support  the 
successful  implementation  of  the  software  across  the  University. 

We  use  five  of  the  myths  proposed  about  e-leaming  to  assess  the  experience  of  out  students.  These  five 
myths  about  e-leaming  and  the  student  experience  (Myths  2,4,5,  6 & 9)  are  first  identified  and  we  assess  how 
they  relate  to  the  perceptions  of  our  student  data.  We  believe  that  by  assessing  the  data  through  the  myths,  we 
can  assess  whether  the  myth  is  tme  in  the  case  of  our  techno-sceptic  approach.  The  remaining  five  myths 
(Myths  1,3, 7,8,  & 10)  relate  to  the  macro  issues  of  e-leaming  in  higher  education  and  as  this  goes  beyond  our 
evaluation  data,  we  do  not  assess  these  myths. 

2.2  Online  Reading  List 

The  Library  at  our  institution  decided  to  implement  an  online  reading  list.  RLO  was  made  available  to 
students  in  September  2012.  The  first  year  of  the  implementation  is  being  referred  to  as  a ‘soft  launch’  where 
the  system  is  made  available  to  students  at  a ‘beta’  stage.  At  this  stage  traditional  paper  reading  lists  are  being 
added  to  the  ‘cloud’  continuously,  until  all  reading  lists  from  across  the  University  are  accessible  to  students. 
Wherever  a lecturer  has  decided  to  use  RLO,  they  inform  their  students  of  its  availability.  The  students  are 
either  given  a link  to  the  open  online  reading  list  or  the  reading  list  is  made  available  in  the  VLE  as  a 
‘Moodle  Block’.  Alternatively  the  students  can  search  for  their  relevant  module  via  the  home  page 
(http://readinglists.city.ac.uk/index.html) 

From  a Library  perspective,  online  reading  lists  can  be  considered  as  a learning  resource  management  and 
as  a delivery  tool  as  reading  lists  are  a cmcial  stepping  stone  between  the  classroom  and  students’ 
independent  learning  (Chad,  2012);  harnessing  new  technology  to  support  efficient  and  effective  use  of 
reading  lists  will  help  students  to  develop  the  information  literacy  skills  needed  to  become  competent  library 
users  and  confident  learners  (Farmer,  Maclean  & Corms,  2012;  Stokes  & Martin,  2008).  Furthermore, 
McCormick  (2006)  points  out  that  reading  lists  produced  by  academics  for  teaching  purposes  are  a valuable 
academic  tool  for  different  purposes,  including:  they  are  a lecturer  teaching  resource,  a student  learning 
resource  and  for  librarians,  a resource  selection  tool. 

In  addition,  Piscioneri  & Fllavac  (2012)  propose  that  issues  related  to  students  reading  lists  at  higher 
education  can  be  approached  from  at  least  three  angles:  skill  development  framework,  such  as  useful  reading 
strategies  (e.g.  skim  reading),  curriculum  development  perspective,  such  as  looking  at  the  design  aspects  of 
the  reading  list,  and  pedagogical  perspective,  which  focuses  on  reading  lists  as  a tool  to  improve  the  student 
education  experience. 
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Taking  into  account  all  the  student’s  perception  of  reading  lists,  a thematic  analysis  shows  that  our 
students  do  very  much  agree  that  reading  lists  are  part  of  their  academic  experience  in  higher  education.  The 
students’  definitions  of  reading  list  can  be  stated  as  follows: 

“A  recent  and  comprehensive  composition  of  the  most  up  to  date  sources  as  chosen  by  academics  and 
given  to  students  in  a compact  form,  which  acts  as  a beneficial  and  important  precursor  to  the  students 
education  experience”. 

Part  of  the  implementation  strategy  was  to  evaluate  the  technology;  in  particular,  we  were  interested  in 
understanding  how  this  e-leaming  tool  affects  the  student  experience.  Due  to  the  changeable  nature  of  the 
software,  it  was  recognised  that  student  feedback  would  support  the  further  development  of  this  e-leaming 
tool. 


2.3  Methodology 

Twenty-one  students  participated  across  three  focus  groups  with  each  focus  group  lasting  approximately  two 
hours.  In  order  to  encourage  a high  participation  rate,  students  from  across  the  University  were  invited  to 
participate  via  Unitemps  and  they  were  paid  for  their  time.  The  students  (twelve  females  and  nine  males) 
were  from  a range  of  study  disciplines,  consisting  of  undergraduates,  postgraduates  (both  Masters  and  PhD), 
and  recent  graduates,  with  all  current  students  being  full  time.  This  method  is  an  example  of  opportunistic 
sampling:  the  number  of  students  participating  from  programmes  varied  depending  on  the  level  of  interest 
from  the  particular  individual  that  accessed  Unitemps  job  website. 

At  the  broadest  level,  focus  groups  are  collective  conversations  or  group  interviews.  Focus  groups  offer 
unique  insights  into  the  possibilities  of  critical  inquiry  as  a deliberate,  dialogical  and  democratic  practice 
(Kamberelis  & Dimitriadis,  2005).  Focus  groups  have  a number  of  advantages  (Farnsworth  & Boon,  2010) 
but  we  felt  the  most  important  advantage  in  using  focus  group  was  that  the  students  discussed  the  topic  of 
reading  lists  whilst  being  part  of  a group  dynamic;  participant  interaction  allowed  students  to  share  held 
tmths  whilst  also  co-constracting  meaning  (Belzile  & Oberg,  2012). 

There  has  been  a considerable  expansion  of  the  use  of  focus  groups  as  a method  of  data  collection  in 
social  science  research  over  the  last  decade  (Peek  & Fothergill,  2009).  As  such,  for  our  purpose,  we  created 
an  atmosphere  that  encourages  spontaneous  contribution  by  a)  setting  the  scene  through  a number  of 
premeditated  considerations,  such  as  layout  of  the  room  and  creating  informality  and  b)  using  space  and  time 
more  flexibly,  such  as  having  group  activities  and  providing  refreshments  (Morgan  et  al,  2002). 

Each  focus  group  was  divided  into  three  parts:  in  part  one,  discussions  among  students  were  facilitated  to 
gather  verbal  evidence  of  students’  perceptions  about  their  experiences  of  reading  lists  and  how  they 
interacted  with  them  in  academic  and  social  environments  for  their  higher  education  degree.  In  part  two,  the 
students  were  shown  a demonstration  of  RLO  then  asked  to  discuss  their  initial  reactions  to  the  online  tool 
and  how  it  could  change  their  student  experience.  At  the  end  of  the  focus  groups,  the  students  were  asked 
complete  an  adopted  version  of  the  PUEU  questionnaire  (Perceived  Usefulness  and  Perceived  Ease  of  Use) 
(Davis,  1989). 

Davies  (1989)  proposed  that  there  are  two  key  determinants  of  user  acceptance:  ‘Perceived  Usefulness’ 
and  ‘Perceived  Ease  of  Use’.  Fie  showed  that  these  two  factors  could  be  measured  through  a reliable  and 
valid  questionnaire,  the  PUEU.  Since  then  user  acceptance  of  technology  has  been  a critical  part  of  the 
Technology  Acceptance  Model  (Chuttur,  2009).  The  original  questionnaire  was  adopted  to  reflect  the  RLO 
and  higher  educational  context. 

2.4  Qualitative  Data:  Perception  of  New  Learning  Technology 

The  students  were  overwhelmingly  positive  about  RLO;  once  they  were  shown  a demonstration  of  the  online 
tool  and  had  the  chance  to  ask  questions,  they  were  in  consensus  that  RLO  will  have  a positive  impact  on 
their  academic  experience. 

"Most  of  the  problems  are  solved  with  this  system.  Like  90%  of  the  problems  you  face  when  you  have  to 
find  a book"  (Focus  Group  3 student) 
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A recent  graduate  from  the  University  summaries  his  impressions  by  saying: 

“I  wish  I’d  had  something  like  this  when  I did  my  Masters”  (Focus  Group  2 student)  whilst  another 
graduate  stated  that:  "this  whole  system  would  have  been  a Godsend  if  I'd  had  it..."  (Focus  Group  3 student) 

The  students  also  gave  specific  reasons  as  to  why  they  thought  that  this  e-leaming  tool  can  improve  their 
academic  experience.  By  far  the  most  important  reason  was  the  look  and  feel  of  the  system.  They  stated  that 
based  on  their  past  experience  of  traditional  paper  based  reading  lists,  they  find  the  RLO  to  be: 

• Systematic:  a tool  that  saves  time,  and  increases  accuracy  of  finding  books.  In  particular,  the 
students  were  very  much  aware  that  the  RLO  reducing  the  steps  to  accessing  resources 

• Efficient:  the  students  recognised  that  the  tool  will  decrease  error  of  finding  the  correct  materials  as 
the  students  click  directly  to  reading  list  material  rather  than  having  to  type  the  information. 

• Immediate:  the  list  can  be  edited  at  any  time  by  the  lecturer  and  the  changes  would  be  visible  by  the 
students  immediately. 

• Accessible:  the  tool  is  online  so  it  can  be  accessed  at  anytime  from  anywhere 

Moreover,  the  students  expressed  that  there  are  specific  functions  of  the  RLO  that  will  improve  their 
student  experience.  These  include: 

• The  interface  was  perceived  to  be  simple  and  easy  to  use 

• The  ability  to  export  RIS  files  for  use  with  referencing  tools 

• RSS  feeds  that  inform  students  when  a new  resource  has  been  added  to  their  module  reading  list  by 
the  academic 

• The  ability  to  annotate  reading  list  and  save  these  annotations  for  later  viewing 

• Interlink  with  other  e-leaming  tools,  such  Moodle 

• External  links  to  buy  the  books  from  online  stores  such  as  Amazon  and  Google  Books 

Notably,  when  discussing  the  multifunction  of  RLO,  a student  said  that  "that's  so  much  more  than  I was 
expecting  to  be  honest"  (Focus  Group  2 student). 

Furthermore,  the  students  were  explicit  that  this  tool  will  increase  their  engagement  with  their  academic 
readings.  The  positive  perception  and  evident  ease  of  use,  led  to  many  of  the  students  stating  that  the 
productivity  of  the  tool  will  motivate  them  to  read  and  access  more  reading  list  material  given  by  their 
lecturers.  One  student  summarised  this  feeling  by  saying: 

“Yeah  it  kind  of  gives  you  more  edge  to  it,  you  know  motivation  to  actually  read  ...  then  when  we’ve 
done  all  this  work  in  one  you  just  go  click  click  and  find  out  something  more  about  if’  (Focus  Group  2 
student).  Whilst  another  student  agreed  that  “you’ve  put  all  that  there  and  it’s  so  much  easier  just  clicking  on 
the  link,  getting  the  book,  it’s  like  in  terms  of  what  I work  through  and  efficient  would  have  been  just  much 
more  higher  if  we  had  something  like  that  a few  years  ago”  (Focus  Group  2 student). 

Finally,  we  also  asked  the  students  to  describe  how  they  feel  about  RLO  in  one  word.  The  students  used 
the  following  terms  to  describe  their  perception: 

Table  1.  Summary  of  one  word  description  students  used  for  RLO 

One  Word  Description 

Useful 

Time-saver 

Streamlined 

Important 

Efficient 

Helpful 

Quick 

Magic 

Easy 
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2.5  Quantitative  Data:  Acceptance  of  New  Learning  Technology 


In  addition  to  the  qualitative  data,  statistical  analysis  shows  that  the  twenty  one  students  that  were  shown  a 
demonstration  of  RLO  perceive  the  RLO  as  both  useful  and  easy  to  use  and  their  results  indicate  that  they 
accept  the  RLO  as  an  information  technology  that  will  support  their  academic  experience.  As  shown  above, 
user  acceptance  of  information  technology  is  important  to  the  adoption  of  the  RLO  and  scores  from  the 
PUEU  could  be  used  to  predict  how  likely  these  students  are  to  accept  the  RLO  as  part  of  their  studies.  The 
mean  scores  for  each  item  are  shown  below: 

Table  2.  Mean  Score  for  each  PUEU  item 


Mean  PUEU  Scores  for  each  item 


N 

Mean 

Using  the  Reading  List 
Online  in  my  studies 
would  enable  me  to 
accomplish  tasks  more 
quickly 

21 

6.48 

Using  the  Reading  List 
Online  would  improve  my 
academic  performance 

21 

5.43 

Using  the  Reading  List 
Online  in  my  studies 
would  increase  my 
productity' 

21 

6.10 

Using  the  Reading  List 
Online  would  enhance  my 
effectiveness  in  my 
studies 

21 

5. 36 

Using  the  Reading  List 
Online  would  make  it 
easierto  do  my  studies 

21 

6.24 

1 would  find  the  Reading 
List  Online  useful  in  my 
studies 

21 

6.1  4 

Learning  to  operate  the 
Reading  List  Online 
would  be  easy  for  me 

21 

6.38 

1 would  find  it  easy  to  get 
the  Reading  List  Online  to 
do  what  1 want  it  to  do 

21 

5.95 

My  interaction  with  the 
Reading  List  Online 
would  be  clear  and 
understandable 

21 

6.05 

1 would  find  the  Reading 
List  Online  to  be  flexible 
to  interact  with 

21 

6.00 

It  would  be  easy  for  me  to 
become  skillful  at  using 
the  Reading  List  Online 

21 

6.38 

1 would  find  the  Reading 
List  Online  easy  to  use 

21 

6.38 

Measurement  of  perceived  usefulness  explores  the  tendency  of  individuals  to  use  a particular  system  or  a 
product,  that  is,  the  extent  to  which  they  believe  it  will  help  and  enhance  their  academic  performance  (Adams 
& McNab,  2012).  PUEU  is  a 12  item  five-point  Likert  scale  structured  questionnaire.  Responses  to  the 
statements  were  coded  into  7 point  scale,  where  one  indicated  unlikely  and  7 likely.  The  higher  the  score  in 
each  individual  scaled  item  would  indicate  more  likeliness  to  agree  with  the  statement.  The  mean  scores  from 
across  the  sample  was  be  used  to  ascertain  how  likely  the  students  are  to  accepting  the  RLO  as  part  of  their 
academic  experience. 

As  can  be  observed,  the  highest  agreement  was  with  the  statement:  “Using  the  Reading  List  Online  in  my 
studies  would  enable  me  to  accomplish  tasks  more  quickly”  with  the  students  scoring  the  RLO  6.48  out  of  7. 

Although  all  the  statements  were  scored  high  (above  5),  the  statement  that  the  students  least  agreed  with 
about  this  e-leaming  tool  was:  “Using  the  Reading  List  Online  would  improve  my  academic  performance” 
with  a mean  score  of  only  5.43. 

Furthermore,  non-paramefric  statistical  correlation  of  the  two  construct  show  that  there  was  a significant 
positive  relationship  between  the  ‘Perceived  Usefulness’  and  ‘Perceived  Ease  of  Use’,  r = .43,  p (one  tailed) 
<0.05. 
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Figure  1.  Scatter  diagram  showing  the  individual  scores  of  each  participants  in  PU  and  EUv 


Table  3.  Pearson’s  correlation  for  PUEU 


Positive  Correlation  between  Percieved  Usefulness  & Percieved  Ease  of  Use 


Percieved 

Usefulness 

Ease  of  Use 

Percieved  Usefulness 

Pearson  Correlation 

1 

.464' 

Ease  of  Use 

Pearson  Correlation 

.464' 

1 

* Correlation  is  significant  at  the  0.05  level  (1-tailed). 


Overall,  the  above  quantitative  data  shows  that  the  students  perceive  the  RLO  to  be  both  useful  and  easy 
to  use.  Furthermore,  the  positive  correlation  between  the  two  constructs  indicates  that  these  students  are 
likely  to  accept  the  RLO  as  a technology  that  will  support  their  academic  experience  at  this  higher  education 
institution. 

2.6  Applying  a Techno-Sceptic  Framework 

Despite  the  overwhelming  positive  reaction  to  RLO,  the  students  showed  a great  level  of  gravitas  and  realism 
when  discussing  the  RLO  as  a replacement  of  traditional  paper  reading  lists.  In  particular  they  shared  their 
opinion  on  the  wider  implications  of  this  online  tool  and  their  experience  as  students. 

Of  the  ten  myths  of  e-leaming  in  higher  education  presented  by  Njenga  and  Fourie,  five  relate  to  the 
student  experience.  These  myths  relate  to  the  student  experience  as  they  are  about  the  pedagogy  of  e-leaming 
and  the  impact  it  has  on  the  students’  relationship  with  the  academia.  These  are: 

Myth  2:  e-Leaming  can  replace  human  interaction 

Myth  4:  Providing  numerous  courses  and  an  abundance  of  information  is  beneficial  and  can  enhance 
learning 

Myth  5:  ICTs  should  become  the  primary  medium  of  learning  in  higher  education 
Myth  6:  Leisure  (including  playing  and  entertaining)  and  learning  are  separate  activities. 

Myth  9:  e-Leaming  will  see  the  demise  of  traditional  campuses 
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From  these  myths,  all  but  myth  6 deal  with  similar  premises:  Njenga  and  Fourie  (2010)  point  out, 
technology  is  just  one  medium,  a means  to  achieve  something  and  not  an  end  in  itself.  According  to  the 
authors,  all  four  of  these  statements  are  fallacies  that  assume  that  e-leaming  can  replace  human  interaction. 
As  much  as  technology  is  important  and  it  has  become  abundant  in  higher  education,  it  is  a myth  that  it  can 
become  the  primary  medium  of  learning  in  higher  education  and  gradually  lead  to  the  demise  of  traditional 
campuses. 

This  techno-scepticism  makes  a particular  note  that  just  because  more  students  can  be  taught  at  the  same 
time  does  not  mean  human  interaction  can  be  replaced.  They  believe  that  this  myth  is  perpetuated  by  the 
constructivist  notion  of  transferring  the  responsibility  of  learning  to  the  learners. 

Our  evaluation  data  would  suggest  that  it  is  indeed  the  case  that  students  are  completely  opposed  to  the 
notion  of  decreasing  human  interaction  with  the  academics.  They  believe  that  human  interaction  is  required 
for  meaningful  student  learning  to  take  place.  In  particular,  the  students’  opinions  about  this  e-leaming  tool 
were: 

• it  should  enhance  face  to  face  learning,  not  aim  to  replace  it; 

• it  has  limitations,  which  should  be  constantly  improved; 

• it  is  only  as  good  as  the  institution  and  academics  makes  it; 

• this  online  reading  list  can  play  an  important  role  in  improving  the  students’  experience. 

These  four  myths  also  relates  to  a more  popular  myth  of  the  ‘Digital  Natives’.  The  digital  native  theory 
(Prensky,  2001),  states  that  students  entering  higher  education  today  are  all  “native  speakers”  of  the  digital 
language  of  computers,  videogames  and  the  Internet.  Flowever,  this  notion  has  been  criticised  (Brown  & 
Czemiewicz,  2010)  as  being  just  a myth.  Based  on  our  student  data,  we  would  also  make  a similar  conclusion 
as  students  had  strong  preference  for  print  format  material.  In  faci  when  asked  for  their  recommendations  for 
improving  the  RLO,  a prominent  answer  across  the  three  focus  groups  were  in  regards  to  improving  the 
printing  function  of  the  reading  lists. 

"It's  more  a question,  but  can  you  print  the  reading  list  easily?. ..That's  good  [that  RLO  allows  for  printing 
of  reading  lists],  just  because  sometimes  online  things  are  really  unfriendly  to  print  so  if  you  want  to  put  it  in 
your  folder  with  your  module  so  you  can  tick  bits  off  you  can't"  (Focus  group  1 student). 

It  becomes  clear  that  the  concept  of  digital  native  is  a complex  and  contradictory.  For  example  a number 
of  students  scan  books  and  then  print  the  scans  rather  than  reading  them  online  as  pdfs.  In  conclusion,  one 
sludenl’s  quote  summaries  the  general  techno-scepticism  trend  across  all  focus  groups: 

“...in  terms  of  reading  lists  I prefer  to  have  the  hard  copy  instead  but  [the  RLO]  would  be  more  useful  as 
it  has  more  functions  so  I can  easily  have  access  to  the  books  and  resources...!  can  find  it  online,  the 
consumer  can  buy  it,  see  how  much  it  costs  and  to  see  whether  it  is  available  in  the  library”  (Focus  group  2 
student). 

Myth  6 is  that  ‘leisure  and  learning  are  separate  activities’.  According  to  the  Njenga  and  Fourie,  it  is  a 
myth  that  for  the  students,  leisure  and  learning  activities  are  separate.  They  argue  that  in  fact  these  two  are 
not  separate,  especially  in  the  application  of  e-leaming  and  that  academics  will  have  to  adopt  technology  to 
meet  the  students’  needs. 

The  discourse  among  our  student  sample  was  that  although  it  is  trae  that  technology  is  pervasive  in  all  our 
lives,  they  still  make  a clear  distinction  between  technology  for  leisure  and  technology  for  learning.  Our 
students  were  clear  that  they  did  actually  differentiate  between  leisure  and  learning  activities  and  as  such  they 
differentiate  between  the  technologies  that  assist  in  them.  The  following  discussion  between  the  students  in 
focus  group  1 is  particularly  revealing: 
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Student  3:  I think  the  problem  with  that  is  it’s  not  adapted  for  study  yet,  because  the  whole  problem 

with  e-reading  and  studying,  it  hasn’t  taken  off  and  it  probably  won’t  for  awhile  until  they  have  the  capability 

for  people  to  start  writing  and  highlighting....  Students  aren’t  really  taking  to  it.  A lot  of  people,  when  they  get 
a PDF  they’ll  print  it  out. 

Student  4:  Some  people  find  it  really  hard  to  do  it  on  a computer,  reading  it  from  . . . 

Student  5:  It’s  just  not  the  same. 

Student  2:  It’s  really  difficult  to  study  on  the  computer.  Right  now  I’m  doing  an  analysis  of  the 

Leveson  Inquiry  so  I have  to  go  through  the  whole  transcript. . .Now,  when  I close  my  eyes  I’m  just  seeing 
transcripts  in  my  head.  It  really,  really  messes  with  your  mind.  That’s  why  books  are  so  essential.  Actually 
picking  up  a hard  text  now  is  like  a relief  now  for  my  brain.  It’s  really,  really,  really  difficult  to  study  on  the 
computer. 

Researcher:  But  as  you  said,  it  could  be  just  that  society  hasn’t  reached  that  point. 

Student  3:  I think  in  a few  years  we’ll  probably  all  be  using  e-readers  to  study. 

Student  4:  Agreed.  Not  our  generation. 

Student  6:  Even  if  I equally  use  my  i-pad  for  uni  as  much  as  I ...  it’s  still  not  going  to  replace  the  fact 

that  I need  to  replace  the  fact  that  I need  to  study  with  books. 


This  is  particularly  surprising  as  all  our  students  were  ‘generation  y’,  the  exact  group  of  people  identified 
by  Njenga  and  Fourie  (2010)  to  be  the  group  that  do  not  distinguish  between  leisure  and  learning  activities. 
They  seem  to  fail  to  take  into  account  the  complex  interaction  of  the  students'  active  use  of  information  as 
opposed  to  its  passive  consumption. 


3.  CONCLUSION 

In  September  2012  the  Library  launched  a new  Online  Reading  List.  While  having  a ‘soft  launch’  of  the  new 
software,  we  took  the  opportunity  to  investigate  the  views  of  students  on  how  reading  lists  relate  to  their 
academic  experience  in  higher  education.  In  particular,  we  wanted  to  understand  the  perception  of  the 
students  towards  the  new  e-leaming  tool. 

The  results  were  clear;  the  students  had  a very  positive  reaction  to  the  online  reading  list.  Moreover,  their 
results  on  the  PUEU  indicated  that  they  accepted  the  technology  as  a tool  that  can  improve  their  academic 
experience  at  the  higher  education  level. 

Flamessing  new  technology  to  support  efficient  and  effective  use  of  reading  lists  will  help  students  to 
develop  the  information  literacy  skills  needed  to  become  competent  library  users  and  confident  learners.  As 
reading  lists  play  a central  part  in  higher  education,  student  learning,  information  literacy  skills  development 
and  academic  achievement,  understanding  the  experiences  and  perceptions  of  students  using  and  learning 
from  online  reading  lists  warrants  this  investigation  from  e-leaming  practitioners,  such  as  that  by  Franklin 
(2012). 

Furthermore,  discussions  in  the  student  focus  groups  went  beyond  just  this  technology  and  touched  into 
epistemology  of  e-leaming.  These  discussions  were  presented  through  a techno-scepticism  lens,  in  particular, 
applying  five  of  Njenga  and  Fourie’s  (2010)  myths  that  relate  to  the  student  experience. 

Our  data  indicates  that  the  students  have  much  more  of  a techno-scepticism  perspective  of  e-leaming  than 
practitioners  sometimes  assume.  The  students  made  it  very  clear  that  although  e-leaming  tools  such  as  RLO 
have  a number  of  advantages  that  improve  their  experience,  it  cannot  replace  human  interaction  of  a campus 
based  University.  As  Njenga  and  Fourie  (2010:  207)  point  out:  “Understanding  the  human  complexities, 
both  of  the  educator  and  learner,  would  therefore  ensure  more  effective  e-leaming  adoption,  and  will  ensure 
that  the  difficulty  of  its  use  is  reduced  if  not  eliminated  altogether.” 

This  research  demonstrates  that  when  educational  institutions  step  back  and  reflect  on  critical  questions 
regarding  the  use  of  technology  in  higher  education,  the  results  may  be  surprising.  There  are  a number  of 
myths  around  e-leaming  in  higher  education  and  understanding  the  student  perspective  adds  our 
philosophical  positioning  and  practical  decisions. 

Finally,  the  authors  acknowledge  that  there  are  a number  limitations  to  the  research,  including:  1) 
background  of  the  students  were  not  random  and  does  not  represent  all  students  experience;  2)  small  sample 
for  the  quantitative  analysis;  3)  the  focus  group  and  questionnaire  observe  only  a snapshot  of  the  immediate 
reaction  on  the  acceptance  and  not  their  reaction  of  using  the  tool  as  part  of  their  studies  in  the  long  term. 
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ABSTRACT 

SMS  delivery  platforms  are  being  increasingly  used  at  the  university  level  to  enhance  student  achievement  as  well  as 
traits  and  attitudes  related  to  the  learning  process.  SMS  delivery  provides  access  to  learning  materials  without  being 
limited  by  space  or  time  and  sophisticated  technological  advances  in  SMS  delivery  have  led  to  enhanced  learner 
motivation,  learner  curiosity,  learner  autonomy,  learner  self-efficacy,  learner  technological  self-confidence  when  learning 
language,  vocabulary  and  concepts.  In  addition  SMS  delivery  of  learning  materials  has  been  shown  to  lead  to  flexible, 
user-friendly,  controlled  and  adaptive  learning  as  reported  by  university  students  in  various  research  studies.  The  present 
study  is  another  in  a series  of  research  studies  designed  to  examine  the  relationship  between  SMS  delivery  and  cognitive 
and  affective  aspects  of  university  level  learning.  In  the  present  study  two  groups  of  first  year  university  students  who 
studied  cultural  concepts  in  an  elective  14  week  long  (semester)  course  were  exposed  to  two  different  modes  of  concept 
delivery.  The  first  group  of  students  received  weekly  lists  of  cultural  concepts  sent  via  SMS  messages  to  their  cell-phones 
and  the  second  group  received  weekly  lists  of  cultural  concepts  sent  via  hardcopy  snail  mail  messages  to  their  homes.  The 
definitions  of  cultural  concepts  studied  by  SMS  and  hardcopy  delivery  strategies  were  identical  and  the  students  received 
25  cultural  concept  definitions  on  a weekly  basis  for  a period  of  14  weeks.  At  the  end  of  this  period  the  students  in  the 
two  groups  were  tested  on  a standardized  cultural  concepts  learner  achievement  test  and  responded  to  a questionnaire  that 
examined  the  levels  of  learner  creativity,  learner  flexibility  and  learner  self-image  as  perceived  by  the  students’  in  the  two 
groups.  Results  of  the  study  indicate  that  there  were  no  significant  differences  between  students  in  the  SMS  delivery 
group  and  those  in  the  hardcopy  delivery  group  on  the  standardized  cultural  concepts  achievement  test.  However,  there 
were  significant  differences  between  the  students  in  the  two  delivery  groups  regarding  their  levels  of  learner  creativity, 
learner  flexibility  and  learner  self-image.  The  students  who  received  cultural  concepts  via  SMS  delivery  were 
characterized  by  significantly  higher  levels  of  learner  creativity,  learner  flexibility  and  learner  self-image  than  their 
counterparts  who  received  lists  of  cultural  concepts  via  hardcopy  messages.  It  appears  that  SMS  delivery  of  cultural 
concepts  enhances  traits  and  attitudinal  variables  such  as  learner  creativity,  learner  flexibility  and  learner  self-image 
which  have  a positive  effect  on  the  learning  experience  Thus  SMS  delivery  of  learning  materials  can  in  fact  become  a 
viable  technological  mobile  delivery  system  in  the  university  learning  process  and  should  be  considered  as  a valid 
pedagogical  tool  in  the  university  learning  process. 


KEYWORDS 

SMS  delivery;  hardcopy  delivery;  learner  achievement;  learner  creativity;  learner  flexibility;  learner  self-image 


1.  INTRODUCTION 

Distance  learning  has  developed  over  the  years  to  overcome  the  limitations  of  traditional  face-to-face 
learning  which  necessitates  the  presence  of  the  student  in  a formal  classroom  setting.  Since  its  inception 
when  distance  learning  was  confined  to  the  delivery  of  learning  material  via  snail  mail,  landline  telephone 
and  radio  broadcasts,  it  has  progressed  through  delivery  systems  such  as  television  broadcasts, 
videoconferencing  and  email,  and  at  present  focuses  on  digital  delivery  systems  such  as  internet  and  mobile 
learning  platforms.  It  should  be  noted  that  almost  all  of  the  above  distance  learning  delivery  platforms  are 
still  in  use  in  different  educational  systems  throughout  the  world  (Katz  & Yablon,  2003). 

After  the  development  of  sophisticated  third  generation  distance  learning  systems  which  include 
interactive  video,  internet,  and  mobile  learning  technologies,  learning  activity  through  the  medium  of  these 
distance  learning  has  been  redefined  to  include  and  focus  on  student  self-learning  (Trentin,  1997).  Mobile 
learning  offers  tuition  that  is  not  bound  by  space  or  time  and  is  especially  characterized  by  flexibility.  In 
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addition  mobile  learning  allow  tutors  to  modify,  reinforce  and  even  model  educational  processes,  thereby 
fulfilling  the  cognitive  as  well  as  affective  needs  and  requirements  of  students  (Wilson  & White  lock,  1997). 

Some  research  studies  (Katz  & Yablon,  2009;  2011;  2012)  have  indicated  that  third  generation  distance 
learning  is  especially  suited  to  higher  education  mainly  because  of  increased  flexibility  due  to  the  mobile 
learning  systems  that  are  increasingly  used  at  present.  Other  studies  have  emphasized  the  importance  of 
student  activity  provided  for  by  current  distance  learning  systems  and  have  indicated  that  the  student  activity 
variable  contributes  significantly  to  improved  student  achievement  (Trentin,  1997). 

Mobile  learning  in  general  and  SMS  based  learning  in  particular  have  advanced  steadily  over  recent  years 
and  have  become  potential  learning  platforms  at  the  university  level.  In  certain  areas,  such  as  the  learning  of 
vocabulary  (Katz  & Yablon,  2009;  2011;  2012)  and  concept  learning  (Katz  & Katz,  2011)  SMS-based 
learning  has  advanced  rapidly  and  is  becoming  an  integral  part  of  the  learning  process  in  many  universities 
throughout  the  world.  Research  studies  have  indicated  that  the  use  of  SMS  as  a delivery  system  for  university 
learning  is  suitable  for  both  cognitive  and  affective  aims  (Divitini,  Haugalokken  & Norevik,  2002;  Gamer, 
Francis  & Wales,  2002;  Prensky,  2005). 

In  the  present  study  learner  achievement  and  the  traits  and  attitudes  of  students  toward  SMS  delivery  of 
learning  content  are  to  be  examined.  More  specifically,  the  study  will  investigate  the  academic  grades 
achieved  by  students  in  cultural  concept  acquisition  as  well  as  some  of  their  traits  and  attitudes,  namely 
learner  creativity,  learner  flexibility  and  learner  self-image,  towards  SMS-based  delivery  of  learning  material 
at  the  university  level. 


2.  SMS  DELIVERY  AND  THE  LEARNING  PROCESS 

Katz  & Yablon  (2009)  found  that  technology  based  learning  in  general  and,  more  specifically,  SMS-based 
learning,  offers  a learning  environment  that  is  especially  characterized  by  flexibility  offered  to  the  learner.  In 
addition  SMS  learning  technology  offers  possibilities  that  include  sophisticated  text  capabilities  that  enhance 
the  learning  process.  Moreover,  learning  is  not  bound  by  space  or  time  and  students  can  choose  to  engage  in 
learning  without  almost  any  limitations  (Dieterle  & Dede,  2006). 

SMS  delivery  of  learning  materials  has  become  a focus  of  research  in  recent  years.  Learning  projects 
based  on  SMS  delivery  of  learning  material  and  initiated  by  several  universities  worldwide  have  indicated 
positive  outcomes  (Divitini,  Flaugalokken  & Norevik,  2002;  Gamer,  Francis  & Wales.,  2002;  Seppala,  2002). 
Stone  & Briggs  (2002)  indicated  how  exercises  were  efficiently  delivered  to  students  via  SMS  based 
messages  and  positively  contributed  to  effective  learning.  Thomas,  Orthober  & Schultz  (2009)  pinpointed  the 
benefits  gained  by  high  school  students  in  their  language  learning  after  receiving  language  related  learning 
materials  via  SMS  messages  sent  to  them  by  their  teachers.  Additional  studies  have  described  how 
vocabulary  transmitted  by  SMS  in  a spaced  and  scheduled  pattern  of  delivery  contributed  to  student 
proficiency  in  English  or  other  languages.  (Kieman  & Aizawa,  2004;  Katz  & Yablon,  2009;  2011;  2012). 
Katz  & Katz  (2011)  indicated  how  SMS  based  delivery  of  learning  content  is  closely  connected  to  effective 
concept  learning. 

2.1  SMS  Delivery  and  Students'  Traits  and  Attitudes 

In  recent  studies  researchers  (Cavus  & Ibrahim,  2009;  Thatcher  & Mooney,  2008;  Thornton  & Flouser,  2008) 
concluded  that  mobile  technology  including  SMS-based  delivery  of  learning  materials  is  increasingly 
welcomed  and  accepted  at  the  tertiary  educational  level  as  a valuable  teaching  tool.  Ismail,  Idms  & Johari 
(2010),  Katz  & Katz  (2011),  Katz  & Yablon  (2009,  2011;  2012),  Moos  & Azavedo  (2009),  Rosli  et  al  (2010) 
as  well  as  Song  (2008)  have  confirmed  the  existence  of  a positive  and  significant  relationship  between  SMS 
delivery  of  learning  material  and  students'  traits  and  attitudes  at  the  university  level. 

Additional  research  studies  have  investigated  the  relationship  between  SMS  delivery  of  learning  material 
in  the  course  of  the  learning  process  and  students'  specific  traits  and  attitudes.  Learner  motivation,  learner 
autonomy,  learner  control  of  the  learning  process,  learner  curiosity,  learner  self-efficacy,  learner 
technological  self-confidence,  and  user  friendliness  have  been  found  to  have  a positive  significant 
relationship  with  SMS  delivery  of  learning  content  (Cavus  & Ibrahim,  2009;  Katz  & Yablon,  2009;  2011; 
2012;  Mainemelis,  Boyatzis  & Kolb,  2002). 
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As  academic  achievement  (Perveen,  2010;  Weng,  Cheong  & Cheong,  2010),  learner  creativity 
(McWilliam  & Dawson,  2008;  Tillander,  2011),  learner  flexibility  (Greener,  2010;  Mainemelis,  Boyatzis  & 
Kolb,  2002)  and  learner  self-image  (Offir  & Aflalo,  2008;  Renes  & Strange,  2011)  are  issues,  traits  and 
attitudes  considered  important  in  the  learning  process,  the  present  study  examines  the  issue  of  learner 
achievement  attained  by  students  when  using  SMS  delivery  for  concept  learning.  In  addition  the  research 
pays  particular  attention  to  the  examination  of  the  relationship  between  learner  creativity,  learner  flexibility 
and  learner  self-image  of  students  on  the  one  hand  and  SMS  delivery  of  cultural  concepts  on  the  other. 

2.2  SMS  Delivery  and  Academic  Achievement 

A number  of  researchers  have  addressed  the  issue  of  technology  based  learning  and  academic  achievement. 
Guzeller  (2012  indicated  that  senior  high  school  students  who  utilized  technology  based  learning  strategies 
achieved  higher  academic  scores  in  tests  that  assessed  language  learning  than  their  counterparts  who  studied 
language  without  technology-based  facilitators.  Efendioglu  (2012)  investigated  the  use  of  technology  in  a 
pre-service  teachers'  training  course  and  its  relationship  with  academic  achievement  of  the  teacher  trainees. 
Results  of  the  study  indicated  that  technology-based  instruction  facilitates  a higher  level  of  academic 
achievement  than  more  traditional  instructional  strategies.  Huffman  & Huffman  (2012)  indicated  that  use  of 
technology  in  the  instruction  of  college  students  enhanced  their  academic  success  and  grades.  On  the  other 
hand  Katz  & Yablon  (2009;  2011;  2012)  found  no  significant  differences  between  the  academic  achievement 
attained  by  first-year  university  students  who  utilized  technology  in  the  learning  process  and  that  attained  by 
their  counterparts  who  studied  without  technology-based  instructional  strategies.  Although  the  relationship 
between  technology-based  instruction  and  academic  achievement  is  not  quite  clear,  this  study  will  investigate 
the  potential  link  between  SMS-based  delivery  of  learning  content  and  academic  achievement  as  the 
confirmation  of  such  a relationship  could  have  major  implications  for  university  learning 

2.3  SMS  Delivery  and  Learner  Creativity 

Black  & Browning  (2011)  reported  that  the  use  of  technology  in  education  leads  to  an  increase  of  learner 
creativity.  Hope  (2010)  indicated  that  creative  learning  is  enhanced  by  innovative  learning  environments  as 
found  in  technology-based  learning.  Lussier  & Achua  (2004)  indicated  that  a major  goal  necessary  in  the 
process  of  strengthening  a learning  organization  is  the  promotion  of  creative  thinking  which  can  be  facilitated 
by  the  use  of  technology.  Antonenko  & Thompson  (2009)  contended  that  the  use  of  technology  in  the 
learning  process  significantly  contributes  to  the  promotion  of  learner  creativity.  Eaglestone  et  al  (2007)  stated 
that  the  use  of  technology  in  learning  can  enhance  creativity  in  the  cognitive  process  and  facilitate  knowledge 
acquisition  as  ell  as  affective  well-being.  Akinwamide  & Adedara  (2012)  confirmed  that  technology  based 
learning  platforms  contribute  significantly  to  the  enhancement  of  learner  creativity,  mainly  because  of  the 
facilitation  of  self-discovery  and  autonomy  inherent  in  the  new  digitalized  pedagogy.  Because  of  the 
confirmed  importance  attached  to  creativity  in  the  technology-based  learning  process,  this  study  will  examine 
the  relationship  between  SMS-based  delivery  of  concept  learning  and  learner  creativity. 

2.4  SMS  Delivery  and  Learner  Flexibility 

Osborne  & Oberski  (2004)  as  well  as  Jeffries  (2005)  contended  that  the  multidimensional  nature  of 
technology  can  effectively  enhance  student  flexibility  in  the  learning  process.  Luppicini  (2006)  indicated  that 
students  who  studied  in  a technology  based  environment  adapt  to  the  environment  and  enhance  their  learning 
flexibility  so  as  to  increase  the  quality  of  their  learning.  Lee  (2007)  concluded  that  technology  based  learning 
sensitizes  the  students  ' awareness  of  others  in  their  learning  environment  and  facilitates  their  ability  to  react 
flexibly  in  the  class  or  group  learning  process.  Sweeny  (2010)  postulated  that  technology  used  in  current 
learning  situations  promotes  learning  flexibility  among  students.  This  learner  flexibility  encourages  students 
to  utilize  various  and  appropriate  technologies  in  their  learning  processes.  In  view  of  the  relationship  between 
technology  and  learner  flexibility  indicated  in  the  abovementioned  studies,  this  research  will  test  the 
connection  between  the  reception  of  learning  material  through  the  medium  of  SMS  messaging  and  learner 
flexibility  of  the  student  recipients. 
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2.5  SMS  Delivery  and  Learner  Self-Image 

Sami  & Pangannaiah  (2006)  investigated  the  relationship  between  the  use  of  technology  in  the  learning 
process  and  the  self-image  of  learners.  They  indicated  that  students  who  used  technology  efficiently  in  the 
library  during  their  learning  process  acquired  a more  positive  learner  self-image  than  those  who  were  less 
able  to  make  efficient  use  of  technology  during  the  time  they  spent  learning  in  the  library.  Tabata  & Johnsmd 
(2008)  confirmed  that  students  who  used  technology  more  effectively  in  their  learning  developed  a more 
positive  self-image  of  themselves  as  learners  than  those  less  able  to  constructively  utilize  technology  in  their 
learning  processes.  Nicolle  & Lou  (2008)  confirmed  that  using  innovative  technological  innovations  in 
learning  enhances  students'  perceptions  of  their  learning  as  well  as  their  learner  self-image.  In  light  of  the 
above  research  findings  this  study  will  investigate  the  link  between  learning  delivered  to  students  via  SMS 
messages  and  their  learner  self-image. 

The  research  model  depicting  the  independent  and  dependent  variables  are  presented  in  Figure  1 . 


Figure  1.  Graphic  Illustration  of  the  Research  Model 


3.  METHOD 


3.1  Sample 

The  research  sample  consisted  of  296  first  year  students  enrolled  in  a 14  week  semester-long  cultural 
concepts  foundation  course  offered  at  one  of  the  seven  chartered  universities  in  Israel.  Students  came  from 
similar  socio-economic  backgrounds  and  all  were  accepted  to  the  Faculty  of  Social  Sciences  at  the  university 
after  attaining  the  university  acceptance  criteria  of  a mean  matriculation  grade  of  at  least  80%  as  well  as  a 
mean  psychometric  university  entrance  score  of  600.  The  students  were  randomly  assigned  to  the  two 
different  research  groups  in  which  they  were  provided  with  lists  of  definitions  of  cultural  concepts  via  two 
alternative  learning  content  delivery  methods.  The  first  group  of  163  students  received  their  cultural  concepts 
via  SMS  delivery  to  their  personal  cell-phones  and  the  second  group  of  133  students  received  their  cultural 
concepts  in  hardcopy  format  by  way  of  regular  snail  mail. 
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3.2  Instruments 

Two  research  questionnaires  were  administered  to  the  students  in  this  research  study.  A standardized  cultural 
concepts  learner  achievement  test  was  administered  to  the  participants  in  order  to  assess  students’  mastery  of 
definitions  of  cultural  concepts  relating  to  Israeli  culture.  The  test  scale  ranged  from  0-100,  the  higher  grades 
indicating  higher  levels  of  achievement  on  the  cultural  concepts  test.  The  second  instrument  administered 
was  a 30  item  Likert  type  scale  response  questionnaire  (students  responded  to  a five  point  scale  with 
l=totally  disagree  and  5=totally  agree)  designed  to  examine  the  students’  perceptions  of  the  trait  and 
attitudinal  research  factors  as  follows:  The  first  factor,  learner  creativity,  contained  ten  items  (Cronbach 
a=0.79),  the  second  factor,  learner  flexibility,  consisted  of  eleven  items  (Cronbach  a=0:83)  and  the  third 
factor,  learner  self-image,  was  made  up  of  nine  items  (Cronbach  a=0.78). 

3.3  Procedure 

Students  who  studied  in  the  Faculty  of  Social  Sciences,  and  were  enrolled  in  the  elective  cultural  concepts 
foundations  course  and  possessed  personal  cell-phones  with  texting  capacity  were  eligible  for  participation  in 
this  study.  Following  the  selection  of  the  students  who  met  the  above  criteria,  they  were  randomly  assigned 
to  the  two  delivery  platform  groups.  Students  in  the  first  group  received  cultural  concepts  via  SMS  messages 
sent  to  their  personal  cell-phones  and  those  in  the  second  group  received  cultural  concepts  in  hardcopy 
format  via  snail  mail.  The  students  in  the  two  groups  were  sent  weekly  lists  that  contained  concise  definitions 
of  the  cultural  concepts  studied  in  the  course,  each  weekly  list  contained  definitions  of  25  new  cultural 
concepts  delivered  via  the  respective  learning  strategies.  Thus  each  of  the  students  received  definitions  of  350 
cultural  concepts  during  the  14  week  long  course.  On  completion  of  the  course  the  students  in  the  two  groups 
were  administered  a standardized  cultural  concepts  learner  achievement  test  in  order  to  asses  their  level  of 
knowledge  of  the  350  cultural  concepts  taught  in  the  course.  In  addition  they  were  administered  the  30-item 
trait  and  attitudinal  questionnaire  which  examined  their  scores  on  the  three  attitudinal  research  factors, 
namely  learner  creativity,  learner  flexibility  and  learner  self-image. 


4.  RESULTS  AND  DISCUSSION 


The  main  aim  of  this  study  was  to  examine  the  efficiency  and  effectiveness  of  two  different  learning  delivery 
platforms,  namely  SMS  based  delivery  and  hardcopy  delivery.  Two  research  questions  were  posed:  the  first 
examined  the  acquisition  by  students  of  knowledge  focusing  on  cultural  concepts  and  the  second  investigated 
students’  attitudes  connected  to  the  two  delivery  strategies.  The  mean  scores  of  each  of  the  attitudinal  factors 
were  standardized  in  order  to  allow  for  a comparison  between  the  factor  scores.  Standardized  means  and 
standard  deviations  of  students’  scores  on  the  achievement  test  and  on  the  attitudinal  factors  are  presented 
below. 

Table  1.  Mean  Scores  and  Standard  Deviations  for  Students  in  SMS  and  Hardcopy  Delivery  Groups  for  Learner 
Achievement,  Learner  Creativity,  Learner  Flexibility  and  Learner  Self-Image 


Research  Variables 

SMS  Delivery  (N=l 63) 

Hardcopy  Delivery  (N=133) 

Mean 

S.D 

Mean 

S.D. 

Learner  Achievement 

84.67 

8.64 

85.32 

9.31 

Learner  Creativity 

3.83 

0.44 

3.44 

0.50 

Learner  Flexibility 

3.48 

0.50 

3.32 

0.60 

Learner  Self-Image 

4.00 

0.66 

3.30 

0.73 

One-way  analyses  of  variance  (ANOVA)  were  conducted  in  order  to  investigate  intergroup  differences  on 
the  four  research  variables.  No  significant  differences  were  found  between  students  in  the  SMS  and  hardcopy 
delivery  groups  on  grades  attained  on  the  standardized  learner  achievement  test.  On  the  other  hand  significant 
differences  between  the  two  delivery  groups  were  yielded  for  learner  creativity  [F(l,292)=37,  p<0.001, 
q^=11.2%];  for  learner  flexibility  [F(l,292)=5.36,  p<0.05,  r]^=L8%];  and  for  learner  self-image 
[F(l,292)=55.26,  p<0.001,  ri^=16%].  Post-hoc  Scheffe  tests,  conducted  to  ascertain  specific  inter-group 
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differences  for  the  three  affective  variables,  indicated  that  students  in  the  SMS  delivery  group  attained  higher 
scores  on  the  three  trait  and  attitudinal  variables  than  students  in  the  hardcopy  delivery  group 

A discriminant  function  analysis  was  conducted  in  order  to  check  the  significance  of  the  results  obtained 
in  the  one-way  analyses  of  variance  and  post-hoc  Scheffe  tests.  The  results  of  the  discriminant  function 
analysis,  indicated  that  for  the  three  attitudinal  variables,  namely  learner  creativity,  learner  flexibility  and 
learner  self-image,  an  average  of  64.33%  of  the  students  were  correctly  classified  as  members  of  their 
respective  learning  groups,  thus  affirming  the  significant  differences  indicated  between  the  SMS  and 
hardcopy  learning  content  delivery  groups  regarding  the  traits  and  attitudes  characterizing  group  members. 

The  research  findings  of  this  study  indicate  that  neither  the  SMS  nor  the  hardcopy  delivery  strategies  held 
any  advantage  regarding  learner  achievement  on  the  standardized  cultural  concepts  achievement  test. 
Students  who  studied  the  350  cultural  concepts  via  SMS  delivery  of  concepts  sent  to  their  personal  cell 
phones  or  by  concepts  delivered  to  them  by  hardcopy  via  snail  mail  attained  similar  grades  on  the 
standardized  learner  achievement  test.  Thus  it  seems  that  different  learning  content  delivery  strategies  do  not 
necessarily  lead  to  differential  academic  achievement.  Although  this  result  contradicts  evidence  presented  by 
Guzeller  (2012  and  Efendioglu  (2012),  it  confirms  similar  results  which  have  indicated  that  academic 
achievement  is  not  conditional  to  type  of  learning  strategies  or  delivery  platforms  used  to  facilitate  the 
learning  process  (Bohlen  & Ferratt,  1993;  Dyer  & Osborne,  1996;  Katz  & Yablon,  2009;  2011;  2012). 

Additional  research  results  clearly  indicate  that  the  two  different  content  delivery  strategies  employed  in 
the  present  study  are  related  to  significantly  differential  levels  of  learner  creativity,  learner  flexibility  and 
learner  self-image.  Scores  attained  by  students  in  the  SMS-based  learning  content  delivery  group  on  the  three 
trait  and  attitudinal  factors  were  significantly  higher  than  those  of  students  in  the  hardcopy  learning  content 
delivery  group.  It  appears  from  the  nature  of  these  results  that  SMS-based  delivery  of  learning  material  is 
linked  to  higher  levels  of  learner  creativity.  This  finding  confirms  similar  indications  regarding  the  unique 
advantages  of  technology  in  enhancing  learner  creativity  as  reported  by  current  researchers  including 
Akinwamide  & Adedara  (2012),  Antonenko  & Thompson  (2009)  and  Black  & Browning  (2011).  Similarly 
the  research  result  that  indicates  that  SMS  delivery  of  learning  content  appears  to  be  more  significantly 
related  to  learner  flexibility  than  delivery  by  hardcopy  via  snail  mail  confirms  the  findings  of  Lee  (2007), 
Luppicini  (2006)  and  Sweeny  (2010)  who  suggested  that  technology  has  the  potential  to  enhance  learner 
flexibility.  In  addition  the  finding  of  the  present  study  that  SMS  delivery  of  learning  content  is  more 
significantly  connected  to  learner  self-image  than  hardcopy  delivery  strengthens  the  findings  of  Nicolle  & 
Lou  (2008),  Sami  & Pangannaiah  (2006)  and  Tabata  & Johnsrud  (2008)  who  reported  that  research  studies 
conducted  by  themselves  indicated  that  technology  based  learning  leads  to  an  enhancement  of  learner  self- 
image. 


5.  CONCLUSION 

The  results  of  the  present  study  indicate  the  potential  of  SMS  messaging  of  relevant  subject  matter  as  a 
positive  delivery  platform  significantly  related  to  trait  and  attitudinal  variables  such  as  learner  creativity, 
learner  flexibility  and  learner  self-image.  It  should  be  noted  that  no  relationship  was  found  between  the  two 
delivery  platforms  and  learner  achievement.  Further  studies  need  to  be  conducted  so  as  to  further  explore  the 
possible  relationship  between  delivery  SMS  and  other  technology  based  learning  content  delivery  platforms 
and  learner  achievement.  Flowever,  even  if  no  significant  relationship  is  found  between  the  delivery 
platforms  and  learner  achievement,  the  fact  that  a significant  relationship  exists  between  SMS  delivery  and 
trait  and  attitudinal  variables  such  as  learner  creativity,  learner  flexibility  and  learner  self-image  as  well  other 
similar  attitudinal  variables  as  reported  in  earlier  studies  indicates  the  potential  of  SMS  delivery  for  the 
university  learning  process..  From  a pedagogical  point  of  view  it  appears  that  cell-phone-based  SMS  learning 
content  delivery  leads  to  positive  attitudes  of  students  as  previously  indicated,  among  others,  by  Katz  & 
Yablon  (2009;  2011;  2012)  and  Song  (2008)  and  should  be  universally  considered  as  a legitimate  and 
positive  learning  strategy  at  the  university  level. 
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ABSTRACT 

In  recent  years  e-portfolios  have  received  a great  deal  of  attention  in  the  educational  discourse  and  it  is  associated  with 
individualization  of  the  learning  process,  competence-oriented  assessment  and  institutional  personnel  development. 

Since  the  beginning  of  the  academic  year  2008/09  students  of  the  Catholic  College  of  Education  Linz  had  the  opportunity 
to  create  and  write  their  own  e-portfolio  in  the  context  of  the  teacher  training.  Here  the  e-portfolio  serves  as  an  instrument 
for  professional  development  in  teaching  by  reflecting  one’s  own  learning  process  and  by  illustrating  the  way  of 
competence  development  as  well  as  an  instrument  for  professional  feedback. 

The  objective  of  this  research  is  to  evaluate  motivational  and  technological  factors  towards  using  e-portfolios  among 
students.  Furthermore  e-portfolios  shall  be  developed  as  a tool  and  a method  in  teacher  education.  Authors  will  address 
the  following  research  questions: 

How  is  the  e-portfolio  assessed  by  students  (attitudes,  benefits,  relevance,  usability,  motivation,  acceptance,  etc.)? 
What  factors  do  acceptance  and  motivation  towards  working  with  e-portfolios  depend  upon? 

Does  the  frequency  of  feedback  influence  acceptance  and  motivation? 

Theoretical  basis  for  the  research  is  an  adapted  version  of  model  of  motivation  and  acceptance  with  technology  for 
working  with  e-portfolios  by  students  in  teacher  training.  It  will  be  validated  and  adapted  through  an 
empirical-quantitative  study  based  on  hypothetical  principles.  In  a second  step,  the  resulting  model  is  undergoing  a partial 
least  square  (PLS)  calculation  - an  analysis  in  order  to  estimate  and  interpret  interrelations  and  causal  effects  and  to 
optimize  the  model  of  our  research. 


KEYWORDS 

E-Leaming,  E-Portfolios,  Motivation,  PLS,  Technology  Acceptance,  Teacher  Training,  Usability 


1.  INTRODUCTION 

With  the  Bologna-compliant  revision  of  curricula  at  colleges  and  universities  and  the  orientation  of  the 
studies  towards  studentical  competences  or  „leamer  centred  approach(es)”  (i.e.  outcome  instead  of  input 
orientation)  more  and  more  e-portfolios  have  found  their  way  into  the  curricula  of  various  training  and 
continuing  education  courses.  They  are  associated  with  individualization  of  the  learning  process,  self- 
organized  learning,  reflection  of  the  learning  process,  but  competency  based  assessment  discussed  with 
institutional  staff  development. 

Barbara  Cambridge,  former  Vice  President  of  the  American  Association  for  Higher  Education  indicated 
already  in  2005  in  her  presentation  at  the  „e-Portfolio  Forum  Austria“F*F  that: 

„Leamers  in  the  21st  Century,  who  are  able  to  describe,  apply  and  evaluate  their  knowledge  and 
skills,  will  progress  in  their  role  as  individuals,  citizens  and  will  perform  better  at  work.  E-Portfolios 
are  a way  of  supporting  learners  through  reflection,  integration,  development,  self-management, 
lifelong  learning  and  social  development.” 

In  many  areas  of  the  teaching  / learning  process  is  not  enough  to  identify  and  grade  results  selectively.  In 
the  specific  case  of  school  practical  studies  within  teacher  education  only  an  individual  learning  process 


' http://eportfolio.salzburgresearch. at/index. php?option=com_content&task=view&id=13&Itemid=41&lang=deH  [1.7.2010] 
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allows  acquiring  „professional”  teaching  behaviour  by  making  the  individual  performance  visible.  Unlike 
traditional  forms  of  testing,  such  as  tests,  exams  or  essays,  assessment  methods  enable  focussing  on  problem 
solving  strategies  and  processes  as  well  as  a greater  insight  into  the  underlying  skills  of  prospective  teachers. 

A portfolio  in  general  and  an  e-portfolio  in  particular  can  be  valuable  tools  for  presenting,  discussing  and 
questioning  developmental  processes.  At  the  Private  University  College  of  Education  Linz  e-portfolios  have 
been  used  for  four  years  in  the  practical  school  studies  with  internal  tests  and  external  reports  based  upon  the 
e-portfolio  management  system  „Mahara”  (cp.  Himpsl  & Baumgartner  2009).  Through  a comprehensive  care 
plan  students  but  also  teachers  receive  maximum  support  when  working  with  e-portfolios.  Even  though  e- 
portfolios  are  not  assessed  per  se  they  are  an  essential  component  in  teacher  training  to  represent  the 
students’  growth  of  competence  during  the  practical  training  of  teaching. 

During  practical  training  of  teaching  (that  takes  place  at  selected  schools)  students  are  accompanied  by 
the  practice  teacher  inside  the  respective  schools  as  well  as  by  the  docents  from  the  College  of  Education. 
Practice  teachers  observe  every  teaching  activity  of  the  students  and  discuss  it  with  them  on  the  very  same 
day.  In  addition,  students  can  join  the  lectures  of  their  practice  teachers.  Docents  from  the  College  of 
Education  observe  the  students’  teaching  activity  about  two  to  three  times  a semester.  They  serve  mainly  as 
didactic  and  methodological  experts  and  assist  students  for  professionalizing  their  teaching  skills.  Through 
feedback  from  the  practice  teacher  and  the  peers  the  students’  teaching  should  be  questioned  and  developed 
continuously. 

Within  that  setting  the  e-portfolio  serves  the  individual  documentation  of  the  students’  learning  process. 
To  use  e-portfolios  throughout  the  complete  teacher  training  process  students  must  be  highly  motivated.  Yet 
it  is  not  clear  how  motivation  as  a condition  of  acceptance  behaviour  and  attitudes  is  functioning  specifically. 
For  a successful  and  sustainable  implementation  of  the  e-portfolios  it  also  needs  to  be  analysed  how 
motivation  as  a key  element  of  the  acceptance  process  can  be  developed. 


2.  THEORETICAL  FRAME 

2.1  Definition  of  „E-Portfolio“ 

For  an  appropriate  definition  of  e-portfolios  we  can  find  in  the  literature  numerous  proposals  along  with  a 
variety  of  different  terms  such  as  reflection,  collection,  competence  and  summary.  Since  the  process  of 
personal  skills  development  in  the  practical  training  of  teachers  is  an  essential  criteria  in  teacher  training,  the 
following  definition  of  Hilzensauer  and  Homung-Prahauser  (2005,  p.4)  was  taken  over  from  Salzburg 
Research  as  a basis  for  this  research  project,  who  defines  as  e-portfolio: 

„a  digital  collection  of  ‘skilfully  produced  work’  (artefacts)  of  a person  purporting  to  document  the 
product  (learning  outcomes)  and  the  process  (learning  path  / growth)  of  their  competence 
development  within  a certain  time  and  for  certain  purposes  and  illustrate  if  the  person  concerned  has 
made  the  selection  of  the  artefacts  itself  and  if  it  is  organized  in  terms  of  the  learning  goal  itself  As 
the  owner  this  person  completely  controls  who,  when  and  to  what  extent  it  is  allowed  to  view 
information  from  the  portfolio.” 

This  definition  emphasizes  both  product-orientation  and  process-orientation  as  important  aspects  of  the 
work  with  an  e-portfolio.  The  key  objective  is  therefore  not  collecting  of  artefacts  but  the  documentation  of 
the  Learning  and  performance -related  progress  of  the  learner  on  cognitive,  social  and  emotional  level.  By 
designing  portfolio-documents  over  a longer  period  of  time  an  increase  of  self-responsibility,  self-control  and 
self-assessment  by  increasing  metacognitive  (self-)reflexivity  of  those  involved  is  triggered  (cp.  Stangl 
2004). 

2.2  Self-Determination  Theory  of  Motivation  (Deci  & Ryan,  2000) 

Motivation  plays  an  important  role  in  the  development  of  a human  being  in  general,  but  for  learning  as  well 
as  for  learning  with  e-portfolios  in  particular.  Two  scientifically  widely  accepted  theoretical  approaches  can 
be  found  which  are  the  „Self-Determination  Theory  of  Motivation”  (SDTM)  and  the  “Attention  Relevance 
Confidence  Satisfaction-Model”  (ARCS),  which  partly  overlap. 
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In  the  self-determination  theory  of  motivation  SDTM  developed  by  Edward  L.  Deci  and  Richard  M.  Ryan 
(2000)  is  assumed  that  people  have  the  innate  (intrinsic)  desire  to  explore  actively  and  to  understand  the 
environment  but  also  involving  themselves  into  social  structures,  to  make  connections  and  expand  their 
skills.  These  innate  tendencies  are  essential  to  Deci  and  Ryan,  down  to  three  basic  psychological  needs,  the 
need  for  autonomy,  the  need  for  competence  and  effectiveness  as  well  as  the  need  to  belong,  to  satisfy.  The 
fulfdlment  of  these  basic  needs  is  a prerequisite  for  a healthy  personality  development  and  the  basis  for  the 
intrinsic  (self-)  motivation: 

„The  findings  have  led  to  the  postulate  of  three  innate  psychological  needs  - competence,  autonomy, 
and  relatedness  - which  when  satisfied  yield  enhanced  self-motivation  and  mental  health  and  when 
thwarted  lead  to  diminished  motivation  and  well-being.”  (Deci  & Ryan,  2000a,  p.  68). 

The  need  for  autonomy  is  more  powerful  than  the  other  two  requirements.  Only  in  a self-directed  and 
informal  learning  environment  one  can  experience  competence  and  social  integration,  leading  to  intrinsic 
learning.  When  missing  one  of  the  three  factors  learners  will  avoid  this  learning  setting  or  it  has  a negative 
influence. 

2.3  The  ARCS  - Model  (Keller  & Kopp,  1987) 

Already  in  the  80’s  of  the  last  century  John  Keller  developed  the  ARCS  model  for  designing  instructions  at 
school.  The  model  provides  strategies  for  a systematic  and  targeted  promotion  of  a student’s  motivation.  It  is 
based  upon  the  differentiation  of  four  main  categories  of  motivation:  attention,  relevance,  confidence  in 
success  and  satisfaction. 


Main  Categories 
Attention 

Relevance 

Confidence 

Satisfaction 


Table  1.  Main  categories  of  the  ARCS-Model 
Task  of  the  Motivation  Design 

Creation  and  maintenance  of  attention  and  interest  of  the 
learner. 

Transferring  the  usefulness  of  the  learning  unit  for  reaching 
personal  goals  and  for  the  satisfaction  of  specific  needs. 
Building  of  a positive  expectation  toward  success  and 
competence  opinion  and  perception  of  one’s  own  control. 

Offer  of  attractive  possibilities  for  action,  rewards,  feedback 
and  opportunities  to  assess  their  own  performance. 


Based  on  the  ARCS  - model  recommendations  for  the  design  of  multimedia  learning  environments  have 
been  developed  (cp.  Niegemann  et  ah,  2008,  p 370). 


2.4  Models  of  Acceptance 

Securing  the  acceptance  of  the  use  of  e-portfolios  in  teacher-training  is  essential  for  a successful 
implementation  of  e-portfolios. 

Simon’s  (2001,  p 87)  „acceptance”  means  a „positive  decision  concerning  of  an  innovation  taken  by  the 
user”  in  contradiction  to  the  concept  of  rejection.  By  innovations  novel  products  or  services  are  understood. 
Muller  & Muller-Boling  established  (1986)  the  distinction  between  attitude  and  behaviour  acceptance.  The 
setting  of  acceptance  includes  an  affective  (motivational-emotional)  and  a cognitive  component  and  is  not 
directly  measurable.  With  the  acceptance  behaviour  the  observable  behaviour  toward  the  innovation  (e.g. 
use)  is  explored.  This  dichotomy  is  accepted  in  many  acceptance  models  (e.g.  technology  acceptance  model 
of  Davis  TAMl  1989,  TAM2  of  Davis  and  Venkatesh  2000,  Dynamic  Acceptance  Model  by  Kollmann  1998, 
Model  for  the  use  of  e-leaming  among  teachers  from  Traxler  (2009).  Biirg  and  Mandl  (2004,  p.  5)  transferred 
this  distinction  of  behaviour  versus  attitude  of  acceptance  into  a model  of  acceptance  of  e-leaming  in 
companies  and  emphasize  the  importance  of  individual  characteristics  plus  characteristics  of  the  learning 
environment.  For  the  empirical  study  of  the  acceptance  of  e-portfolio  work,  this  notion  of  acceptance  is  also 
useful  due  to  the  common  focus  on  the  use  of  acceptance  of  IT  systems. 
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2.5  Motivational  and  Technological  Factors  of  e-portfolio  Usage 


Based  on  the  self-determination  theory  of  Deci  and  Ryan  (2000),  the  ARCS-model  of  Keller  and  Kopp 
(1987),  on  the  work  of  Biirg  and  Mandl  (2004,  2005)  and  in  the  acceptanee  model  for  e-leaming  as  well  as  a 
model  for  the  integration  of  acceptanee  motivation  in  media-based  courses  by  Buck  (2006)  we  were  able  to 
derive  personal  factors  versus  contextual  factors  in  the  acceptance  of  working  with  e-portfolios.  These  factors 
were  checked  on  motivational  aspects  in  the  sense  of  self-determination  theory  and  the  ARCS  model.  The 
following  diagram  illustrates  these  relationships  in  an  extended  model  for  motivation  and  acceptance  of 
working  with  e-portfolios.  Factors  that  could  have  influence  on  the  motivation  in  the  sense  of  self- 
determination  theory  („S”)  and  the  ARCS  model  („A”)  were  characterized. 


both  = i&iiAi 


Figure  1 . Model  of  motivation  and  acceptance  toward  working  with  an  E-Portfolio  (own  figure  based  upon  the  models  of 

Biirg  & Mandl,  2004,  combined  with  Buck,  2006) 
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From  the  model,  the  following  research  questions  can  be  derived: 

1.  To  what  extent  there  is  a relationship  between  the  collected  personal  characteristics  and  the  motivation 
toward  working  with  e-portfolios? 

2.  To  what  extent  there  is  a correlation  between  the  assessment  of  the  contextual  factors  and  the  motivation 
to  work  with  e-portfolios? 

3.  To  what  extent  there  is  a relationship  between  motivation  and  attitude? 

4.  To  what  extent  there  is  a link  between  behaviour  and  attitudes? 


3.  METHOD  AND  RESEARCH  DESIGN 

To  answer  the  research  questions  based  on  the  presented  „concept  of  acceptance  and  motivation  of  e- 
portfolio  work”  field  experiment  with  two  quantitative  (written)  surveys  was  used.  Using  this  design  the 
hypotheses  could  be  tested.  Below  is  a structural  overview  of  the  chosen  research  design: 


Explanation: 

T... 

T reatment-Group 

C... 

Control-Group 

O,... 

Survey  1 

O2... 

Survey  2 

X... 

Intervention 

Figure  2.  Empirical  Research  Design 

Two  non- randomized  groups  were  formed,  one  treatment  group  (T)  and  one  control  group  (C).  Members 
of  the  treatment  group  should  have  reported  their  personal  progress,  reflections,  sketches,  etc.  continuously 
for  the  practice  teacher  by  using  the  e-portfolio  and  also  got  amplified  feedback  from  the  practice  teacher  (at 
least  once  a month).  Members  of  the  control  group  could  decide  independently  if  and  when  they  opened  their 
e-portfolios  for  the  respective  practice  teacher.  Those  received  feedback  only  at  the  end  of  the  semester, 
either  orally  or  through  a note  in  the  e-portfolio.  Both,  the  treatment  as  well  as  the  control  group  were 
monitored  by  using  a pre  (Ol)  and  a post  (02)  test  survey. 

The  quantitative  data  collection  was  realized  as  a written  survey  in  web-based  form  at  the  beginning  of 
the  semester  in  October  2011  (the  first  survey)  and  at  the  end  of  the  semester,  conducted  in  February  2012 
(the  2nd  survey).  The  students  received  the  link  to  the  survey  via  e-mail  and  were  asked  to  participate. 
Simultaneously  they  were  informed  that  the  survey  would  be  completely  anonymous.  As  instrument  for  the 
survey  the  professional  standard  software  for  online  surveys  EFS  Survey  (www.unipark.info)  was  used. 

The  following  table  provides  an  overview  of  the  participants  of  the  survey,  who  completed  the  entire 
questionnaire: 


Table  2.  Sample 


Survey 

female 

male 

total 

Nf 

% 

N.„ 

% 

E 

1 (2011/10) 

54 

80,60% 

13 

19,40% 

67 

2 (2012/02) 

54 

78,30% 

15 

21,70% 

69 

Participants  in  both  surveys 

52 

81,25  % 

12 

18,75% 

64 
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The  response  rate  of  the  partieipants  that  partieipated  in  both  surveys  and  filled  in  both  questionnaires 
eompletely  was  71%. 

Based  from  theoretieal  eonsiderations  and  researeh  hypotheses  for  the  ereation  of  the  two  questionnaires 
validated  survey  instruments  were  adopted  from  the  field  of  teaehing  and  learning  researeh  and  eheeked  on 
its  applieability  to  our  researeh  projeets.  The  following  instruments  were  applied: 

Questionnaire  on  the  „Content  differentiated  collection  of  computer-based  settings”  (abbreviated 
FIDEC) 

and  Questionnaire  „Confidence  in  dealing  with  computers  and  computer  applications”  (COMA)  of  the 
“Inventory  of  computer  education”  (INCOBI-R)  by  Naumann,  Riehter  and  Florz  (2010)^ 

Questionnaire  of  ,ficademic  self-efficacy”  of  Jerusalem  and  Satow  (1999)^ 

,Jntrinsic  Motivation  Inventory“  (IMI)  von  Deei  und  Ryan  (1990)"' 

„Short  scale  for  intrinsic  motivation“  KIM  (2009)^ 

Items  from  the  ,Jiesearch  Report  148“  by  Kopp,  Balk  and  Mandl  (2002)  and 
,Jiesearch  Report  161”  by  Kopp,  Dvorak  and  Mandl  (2003)'" 

„System  Usability  Scale”  (SUS)  by  Brooke  (1986)’ 

Missing  questions  were  derived  from  the  theoretieal  eonsiderations.  In  this  study,  a five-level  rating  seale 
with  the  levels  -2  (strongly  disagree)  to  -1-2  (strongly  aeeurately)  was  used. 


4.  RESULTS 

4.1  Correlation  Studies 

In  a first  step  bi- variate  eorrelations  were  ealeulated  as  presented  in  table  3.  Obviously  there  is  a highly 
signifieant  eorrelation  for  both,  personal  faetors  as  well  as  eontextual  faetors,  whereas  personal  faetors 
eontain  soeial,  eognitive -instrumental  and  motivational-emotional  eomponents  and  the  eontextual  faetors 
eover  the  organizational  eontext  plus  the  teehnieal  eharaeteristies  of  the  E-Portfolio. 

Table  3.  Bi- variate  correlations 


Motivation  r^ 

Attitude  r^ 

Gender 

,079 

,128 

Personal  Factors 

,658“ 

,553“ 

Social  Factors 

,677" 

,574" 

Subjective  Norm 

,659“ 

,591“ 

Perceived  Support  (N=45) 

,539“ 

,494“ 

cognitive-instrumental  factors 

,768" 

,645" 

subjective  learning  success 

,598“ 

,495“ 

School-related  self-efficacy 

,045 

,101 

Perceived  Behavioural  Control 

,602“ 

,585“ 

Preliminary  Computer  Knowledge 

-,049 

-,034 

^ Padagogische  Psychologie,  24  (1),  2010,  Hans  Huber,  Bern,  pp  23-37. 

^ http://userpage.fu-berlin.de/~health/self/skalendoku_selbstwirksame_schulen.pdf  [1.7.2012] 
''  http://selfdeterminationtheory.org/questionnaires  [29.12.2012] 

^ http://www.ipn.uni-kiel.de/zfdn/pdf/15_Wilde.pdf  [15.8.2012] 

* http://epub.ub.uni-muenchen.de/273  , http://epub.ub.uni-muenchen.de/256  [12.12.2012] 

''  http://www.measuringusability.com/sus.php  [29.12.2012] 
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Perceived  usefulness 
motivational-emotional  Factors 

,730“ 

,573“ 

Attitude  towards  Computers 

,159 

,150 

Context  factors 

,607“ 

,605“ 

Organizational  context 

,67  r 

,529” 

Relevance 

,731“ 

,592“ 

Information 

,224 

,186 

Support 

-,213 

-,254 

Characteristics  of  the  E-Portfolio  System 

,572“ 

,5sr 

Quality  of  Output 

,498“ 

,610“ 

Usability 

,561“ 

,569“ 

Attitude 

* 

o 

00 

Behavioural  acceptance  (usage) 

,506“ 

,512“ 

Personal  + contextual  Factors 

,639“ 

,629“ 

Ts  = Spearman  correlation  coefficient  - Note:  * p < .05,  **p  < .01  (two  sided  significance  test) 
rs  ^-0,1  minor  effect,  r^  >+-0,3  medium  effect,  r^  >+-0,5  strong  effect 


Behavioural  acceptance  r^ 

Attitude 

,512" 

For  the  following  assumptions  very  high  correlations  had  been  confirmed  as  significant: 

The  subjective  norm  and  the  type  of  care  influence  the  motivation  to  work  with  e-portfolios. 

The  higher  the  subjective  learning  success,  the  perceived  behavioural  control  as  well  as  the  perceived 
benefits  are,  the  greater  the  motivation  to  work  with  e-portfolios  will  be. 

The  perceived  relevance  of  the  curriculum,  the  quality  of  output  and  the  perceived  usability  of  Mahara 
have  a positive  effect  on  the  motivation  for  e-portfolio  work. 

Motivation  and  attitude  of  acceptance  are  closely  related.  The  higher  the  motivation  to  work  with  e- 
portfolio  is  estimated,  the  higher  the  attitude  will  be. 

As  in  other  studies  also  in  this  study  a relationship  between  attitude  and  actual  usage  have  been 
confirmed. 

Interestingly,  no  significant  correlation  between  gender,  school-related  self-efficacy,  computer  technical 
knowledge,  the  IT-affinity,  information  about  the  e-portfolio  system  as  well  as  the  support  and  motivation  to 
work  with  e-portfolios  has  been  found. 

In  the  following,  the  resulting  model  for  motivation  and  acceptance  of  working  with  e-portfolios  in  the 
school  of  practical  studies  teacher  / internal  training  is  presented  in  a summary.  Figure  3 shows  the  adapted 
model  for  motivation  and  acceptance  of  working  with  e-portfolios  in  the  school  practical  studies,  which 
represents  the  relationships  between  each  factor  and  the  motivation  and  acceptance 

Yet  it  does  not  allow  any  statements  about  causal  effect  relationships  or  an  overall  assessment  of  the 
model.  Only  by  using  a structural  equation  analysis  (also  called  „causal  analysis")  cause-effect  relationships 
were  analysed  and  interpreted  in  our  model.  As  a method,  a variance-based  causal  analysis,  based  on  the 
calculation  of  partial  least  squares  (PLS)  has  been  used.  In  contrast  to  the  analysis  of  covariance-structures  it 
requires  no  distributional  assumptions  and  can  also  be  carried  out  using  small  samples.  It  includes  both 
formative  and  reflexive  measurement  models.  With  the  free  software  SmartPLS  we  have  found  a user- 
friendly  tool  that  provides  extensive  calculations  for  the  assessment  of  the  overall  model. 
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For  the  quality  evaluation  of  structural  equation  model  can  be  stated  that  with  respect  to  reliability  and 
validity  good  results  have  been  achieved  and  that  the  operationalization  of  the  latent  variables  succeeded  in 
the  overall  model.  Much  of  the  relations  between  the  influencing  factors  and  the  declared  impact  could  be 
confirmed.  Only  the  direct  relationship  of  the  factors  for  attitude  had  to  be  removed  from  the  model. 
Furthermore  interesting  is  that  the  actual  use  (i.e.  the  behavioural  acceptance)  of  the  e-portfolio  still  has  to 
depend  on  other  factors  that  were  not  considered  in  this  model.  One  possible  factor  could  be  the  available 
time  that  students  were  able  the  students  spend. 


Figure  3.  Resulting  model  of  motivation  and  acceptance  of  e-portfolios 


5.  CONCLUSION 

In  conclusion,  it  can  be  stated  that  motivational  and  technological  factors  as  well  as  their  impact  and 
relationships  were  analysed  systematically  between  the  proposed  latent  variables.  It  has  been  found  that  the 
model  used  basically  provides  a valid  and  reliable  assessment  of  motivational  and  technological  aspects  of 
working  with  e-portfolios.  For  the  interpretation  of  the  results  it  must  be  taken  into  account  that  the  sample  of 
64  subjects  is  a somewhat  small  and  eventually  the  results  and  have  a limited  scope  only.  In  addition,  in  this 
study,  no  effect  on  the  nature  and  quality  of  the  responses  was  made  to  the  e-portfolio  entries,  but  only  a 
minimum  level  of  feedback  from  the  process-oriented  group  has  been  defined.  For  future  research  it  would 
therefore  be  interesting  to  set  priorities  in  the  field  of  qualitative  feedback  here  and  investigate  further 
development  steps.  In  addition,  the  results  presented  here  should  be  reviewed  with  a representative  sample. 

In  summary,  the  following  five  recommendations  for  the  use  of  e-portfolios  in  the  school  of  practical 
studies  of  teacher  can  be  derived  from  the  empirical  study: 

1 . Create  a positive  expectation  for  working  with  e-portfolios  with  students. 

2.  Give  good  and  consistent  support  during  the  work  phase  and  appreciative,  positive  feedback. 

3.  Focus  on  relevance  and  usefulness  for  future  teaching. 

4.  Offer  templates  and  views  but  still  leave  most  freedom  in  the  design  of  e-portfolios. 

5.  Carefully  select  the  e-portfolio  system. 
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ABSTRACT 

Social  media  use  has  become  increasingly  embedded  in  everyday  life.  Among  various  social  media,  Facebook  is  a highly 
interactive  virtual  social  communication  tool  and  it  has  become  increasingly  popular  on  college  campuses.  Research 
results  have  indicated  that  Facebook  could  be  an  effective  platform  for  informal  learning.  However,  most  of  current 
studies  in  the  educational  use  of  social  media  seem  to  pay  more  attention  on  college  students  or  adults.  This  study, 
however,  explored  the  effects  of  Facebook  tutoring  on  learning  English  as  a second  language  for  children.  60  elementary 
students  from  a supplementary  English  tutoring  program  were  evenly  divided  into  Eacebook  tutoring  group  and 
traditional  classroom  teaching  group.  Instructional  strategies  of  communication,  collaboration,  and  information  sharing 
were  given  to  both  groups.  The  results  indicated  that  children  participated  the  Eacebook  tutoring  generally  exhibited  a 
positive  attitude  toward  this  new  type  of  learning  experience.  The  learning  achievement  for  the  Eacebook  group  was  also 
significantly  higher  than  the  traditional  classroom  group.  It  is  concluded  that  providing  with  proper  strategies,  both 
student’s  learning  attitude  and  learning  achievement  were  positively  affected  by  the  Facebook  tutoring. 


KEYWORDS 

Facebook,  learning  English  as  a second  language,  social  network,  supplementary  tutoring 


1.  INTRODUCTION 

Recently,  people  frequently  use  the  Internet  as  a social  medium  to  interact  with  each  other  and  expand  their 
social  circles,  share  information  and  experiences,  and  organize  communities  and  activities  (Bourlard, 
Krishnamurthy,  Sun,  Zhao,  & Liu,  2012).  Social  media  provide  multiple  dimensions  for  developing  creative 
learning  strategies  that  allow  students  to  connect  formal  and  informal  learning  settings.  Students  can  find 
like-minded  people  and  organize  informal  knowledge  exchange  for  educational  purposes  (Wodzicki, 
Schwammlein  & Moskaliuk,  2012).  The  use  of  social  media  applications  in  teaching  and  learning  has 
garnered  substantial  interest  among  educators  (Cain  & Policastri,  201 1). 

Web-based  social  networking  platforms  such  as  Friendster,  MySpace,  and  Facebook  offer  users 
communication  and  interaction  channels  to  link  with  friends,  family,  and  community.  Among  them, 
Facebook  has  become  increasingly  popular  on  college  campuses,. Any  Facebook  user  can  easily  search  and 
view  any  user’s  Facebook  page  through  the  Facebook  network  (Mazer,  Murphy  & Simonds,  2007). 

Most  current  studies  in  educational  use  of  social  media  seem  to  pay  more  attention  on  college  students  or 
adults.  However,  many  younger  children  have  begun  to  bring  social  networks  into  their  daily  life.  They  may 
be  a substantial  group  with  the  potential  to  learn  with  social  networks  as  children  rely  on  interaction  with 
others  who  share  their  interests  in  new  experiences. 

Children  are  curious,  active  and  creative  while  learning  language  (Lin,  2008).  Blattner  & Fiori(2009) 
indicated  that  various  usages  of  Facebook  can  be  integrated  in  foreign  language  courses.  Children  could 
experience  authentic  language  interactions  and  develop  socio-pragmatic  awareness  (e.g.,  language  use  in 
specific  contexts,  relationship  building...),  which  are  often  absence  in  textbooks.  In  Taiwan,  such  notions  are 
particular  interest  since  learning  a second  language,  especially  English,  is  considered  to  be  the  most 
important  supplementary  work  after  the  regular  classes  for  many  school  children. 
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This  paper  provides  a successful  example  of  learning  English  as  a second  language  via  Facebook  for 
Taiwanese  children.  We  believe  the  promising  results  found  in  this  study  could  draw  better  attention  to  the 
issue  of  learning  a second  language  via  social  network  interactions. 


2.  EDUCATIONAL  USE  OE  EACEBOOK 

In  recent  years,  social  networks  have  been  widely  accepted  as  efficient  platforms  for  academic 
communications,  especially  on  university  campuses.  According  to  Blattner  & Fiori  (2009),  Facebook  is  the 
largest  social  network  that  boasts  more  than  100  million  members,  and  it  is  one  of  the  fastest-growing  and 
best-known  sites  on  the  Internet.  As  McFoughlin  and  Fee  (2008)  pointed  out,  Facebook  has  multiple 
functions  of  communications,  information  sharing,  and  collaborative  construction  and  modification. 

Although  early  in  the  Facebook  appeared,  the  main  reason  for  students  using  Facebook  was  keeping 
contact  with  friends  (Ellison,  Steinfield,  and  Fampe,  2007),  Wodzicki,  Schwammlein  and  Moskaluik  (2012) 
later  discovered  that  students  would  be  in  favor  of  exchanging  informal  experiences  if  a collaborative  and 
sharing  environment  is  embedded  in  the  Facebook.  Roblyer,  McDaniel,  Webb,  Herman,  and  Witty  (2010) 
found  that  in  compare  with  traditional  face-to-face  interaction,  students  prefer  to  communicate  with  teachers 
via  Facebook.  This  may  reduce  an  immediate  embarrassment.  Cain  and  Policastri  (2011)  claimed  that 
Facebook  provides  an  informal  learning  environment  for  presenting  contemporary  topics  and  the  thoughts  of 
guest  experts  not  affiliated  with  a college  or  school,  thereby  exposing  students  to  relevant  “real  world”  issues. 

Mazman  and  Usluel  (2010)  recognize  Facebook  has  three  types  of  educational  functions:  communication, 
collaboration  and  resource/material  sharing.  Communication  consist  of  activities  such  as  enabling 
communication  among  students  and  their  instructors,  facilitating  class  discussions,  delivery  of  homework  and 
assignments  by  teachers,  informing  about  resources  and  links  related  to  courses;  consist  of  activities  such  as 
people's  joining  to  academic  groups  related  to  their  schools,  departments  or  classes  and  carrying  on  group 
works  by  sharing  homework,  projects,  and  ideas;  collaboration  consists  of  activities  such  as  exchanging 
multimedia  resources,  videos,  audio  materials,  animated  videos,  resources  and  documents. 


3.  SOCIAL  NETWORKING  SITE  AND  LANGUAGE  LEARNING 

Harrison  and  Thomas  (2009)  recognized  that  web  2.0  technologies  have  been  advanced  as  potentially 
transformative  in  the  area  of  foreign  language  learning.  They  believed  that  web  2.0  technologies  are 
presenting  new  opportunities  for  developing  diverse  online  learning  environments  and  enhancing 
interactivity,  participation  and  feedback  between  students,  their  peer  groups  and  teachers.  They  further 
argued  that  language  learning  is  not  only  how  learners  develop  their  language  skills,  but  also  how  the 
learning  process  impacts  on  their  overall  personal  development  in  terms  of  cultural  values  and  beliefs, 
something  that  has  been  generally  overlooked  in  computer-assisted  language  learning  research  to  date. 

Blattner  and  Fiori  (2009)  indicated  that  Facebook  could  be  an  effective  platform  for  language 
interactions.  Godwin-Jones  (2008)  also  believed  that  Facebook  could  benefit  interpersonal  communications. 
They  believed  that  unique  functions  of  Facebook  provide  a constructive  learning  environment  while 
maintaining  a certain  degree  of  privacy  and  safety. 

Some  evidences  have  been  established  for  the  fact  that  Facebook  helped  language  learning.  Ajjan  & 
Hartshome  (2008)  recognized  that  active  users  could  improve  their  communication  and  writing  skills  through 
virtual  social  interactions  with  Facebook.  Mill  (2009)  discovered  that  language  learning  with  Facebook  was 
effective  as  virtual  social  activities  help  students  develop  better  verbal  communication  competences  in  the 
real  world.  Moreover,  Shin  (2011)  used  Facebook  as  the  platform  to  perform  peer  evaluation.  The  result 
indicated  that  peer  evaluations  at  Facebook  significantly  improved  student’s  learning  interest  and  learning 
motivation.  These  positive  results  all  supported  that  social  interactions  via  Facebook  affect  the  outcomes  of 
language  learning. 
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4.  LEARNING  ENGLISH  AS  A SECOND  LANGUAGE  IN  TAIWAN 

English  is  the  most  popular  language  in  the  world.  Most  countries  take  English  as  second  language.  Private 
supplementary  tutoring  of  English  has  a long  history  in  parts  of  East  Asia  (including  Japan,  Hong  Kong, 
South  Korea  and  Taiwan).  In  Taiwan,  many  Taiwanese  parents  send  their  children  to  language  cram  schools 
to  learn  English  in  order  to  get  a competitive  educational  advantage.  Most  of  the  teachers  at  cram  schools  use 
their  own  teaching  materials.  Some  teachers  are  emphasized  their  textbooks  are  very  effective.  Many  cram 
schools  treated  English  as  a formal  learning  subject  and  mainly  focusing  on  reviewing  school  textbooks 
therefore  students  had  no  time  to  practice  communicating  each  other  with  English  (Su,  2011).  Promising 
development  is  that  some  of  the  supplementary  tutoring  program  at  cram  schools  have  recently  employed 
social  network  as  the  communication  and  teaching  platform  (Suchiao  and  Yachin,  2012). 


5.  METHODS 

An  experiment  was  done  to  examine  the  effects  of  Eacebook-facilitated  English  tutoring.  The  instructional 
context  was  a series  of  home  assignments  for  an  after -class  remediation  program  in  which  activities  requiring 
peer-to-peer  and  teacher-to-peer  communications  via  Eacebook  were  designed.  Experimental  subjects  were 
divided  into  Eacebook  communication  group  and  traditional  classroom  group.  After  the  experiment,  a 
learning  motivation  questionnaire  and  an  achievement  test  were  given  to  collect  data.  The  research  questions 
were  as  follows: 

1.  What  are  the  effects  of  Eacebook-facilitated  tutoring  on  student’s  learning  motivation? 

2.  What  are  the  effects  of  Eacebook-facilitated  tutoring  on  student’s  learning  achievement? 

3.  How  do  the  students  learn  with  the  Eacebook? 

5.1  Subjects 

Subjects  were  60  elementary  school  children  in  an  English  supplementary  tutoring  program  in  Taipei  City  of 
Taiwan.  They  were  under  age  from  9 to  12  and  were  randomly  assign  to  the  experimental  group  (Eacebook- 
facilitated  group)  and  control  group  (traditional  classroom  teaching  group).  Before  the  experiment,  subjects 
in  the  experimental  group  all  had  Internet  connection  at  home.  The  experiment  lasted  for  6 weeks. 

5.2  The  Instructional  Design 

A six-week  instructional  material  was  designed  with  two  major  emphases:  vocabulary  and  grammar.  The 
same  material  was  applied  for  both  experimental  group  and  control  group.  Each  week,  an  independent  topic 
was  taught  with  a variety  of  instructional  strategies  including  discussions,  explorations,  and  information 
searching/sharing.  The  experimental  activities  were  executed  upon  the  reinforcement  section,  in  which  the 
fallible  parts  of  the  material  were  reviewed.  Eor  the  experimental  group,  Eacebook  was  used  as  the 
communication  platform  in  which  students  discussing  and  collaborating  to  achieve  reinforcement  learning 
tasks,  as  well  as  sharing  each  own  resource  with  other  participants.  Eor  the  control  group,  traditional  drill  and 
practices,  group  discussions,  questions  and  answers  were  performed  in  the  classroom.  The  major  difference 
between  the  two  groups  was  that  for  the  control  group,  there  was  no  Internet  activity  involved.  Table  1 
describes  the  experimental  process  for  each  group  and  Eigure  1 displays  the  sample  Eacebook  learning 
screen. 


Table  1.  Learning  activities 


Activity 

Experimental  group  (via  Eacebook) 

Control  Group  (in  classroom) 

Communication 

Teacher-student  dialog 
Student-student  dialog 

Teacher’s  lecture 
Students  raise  questions 

Collaboration 

Teacher  announce  discussion  topics 
Student  online  discussions 

Teacher  announce  discussion  topics 
Students  discussions  in  groups 

Sharing  resource 

Student  uploading  assignment 
Teacher/student  feedback 

Teacher  hand  out  remediation 
material 
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Figure  1.  Sample  Facebook  learning  screen 

5.3  The  Research  Instrument 


In  order  to  collect  data  for  statistical  analyses,  two  major  instruments-  a learning  motivation  questionnaire 
and  an  English  achievement  test  were  designed. 

5.3.1  Learning  Motivation  Questionnaire 

The  questionnaire  was  developed  based  on  keller’s  (1983)  IMMS  (Instructional  Material  Motivational 
Survey)  and  Pintrich’s  (1989)  MSLQ  (Motivated  Strategies  for  Learning  Questionnaire).  It  has  two  factors 
and  six  phases.  The  following  table  displays  these  factors  and  phases  in  detail. 
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Table  2.  Content  of  learning  motivation  questionnaire 


Factor 

Phase 

Sample  question 

Personal  (Internal) 

Interest  of  learning 
Professional  development 
Problem-solving 

I feel  interest  in  learning  English 
I can  spell  more  words  and  use  correct  grammar 
I can  solve  the  problem  that  teacher  raised 

Environmental 

Social  learning 
Web  learning 

I like  to  share  my  information  with  others 
I like  to  find  answers  from  the  Internet 

(External) 

Self-paced  learning 

I like  to  learn  with  my  own  pace 

The  questionnaire  was  consulted  with  two  elementary  school  English  teachers.  Modifications  were  made 
as  evaluator’s  suggestions.  The  revised  questionnaire  was  then  given  to  30  students  who  did  not  attend  the 
experiment  to  ensure  the  reliability.  The  overall  reliability  for  the  personal  factor  was  .758  and  environmental 
factor  was  .761.  These  reliabilities  were  considered  to  be  fairly  acceptable. 

5.3.2  English  Learning  Achievement  Test 

According  to  the  instructional  design,  student’s  learning  achievements  on  vocabulary  and  grammar  were 
tested.  The  test  consists  of  spelling,  fill-in-blank,  and  short-answer  type  of  questions.  The  test  was  given  to 
the  same  30  students  to  verify  the  reliability.  The  result  was  that  the  overall  reliability  of  the  test  was  .729, 
indicating  the  test  had  very  good  reliability. 


6.  RESULTS 

This  study  employed  a single-test  true  experiment.  Subjects  were  randomly  assigned  to  experimental  and 
control  group.  The  experimental  context  was  in  a cram  school  in  Taipei  City  of  Taiwan  in  which 
supplementary  tutoring  was  provided  to  enhance  regular  school  education.  30  students  were  assigned  to  each 
group.  The  experimental  group  utilized  Facebook  as  virtual  communication,  collaboration,  and  sharing 
platform,  while  the  control  group  performed  the  similar  instructional  activities  in  a traditional  classroom.  The 
instructional  materials  were  designed  for  remediation  and  practice.  A learning  motivation  questionnaire  and 
an  achievement  test  were  given  after  6 weeks  of  experiment.  Following  sections  discuss  the  results  of  the 
data  analyses. 

6.1  Facebook  Effects  Regarding  Student’s  Learning  Motivation 

Motivational  data  were  collected  by  the  learning  motivation  questionnaire.  Learning  motivation  divided  into 
two  categories:  personal  (internal)  motivation  and  environmental  (external)  motivation.  A dependent  t-test 
was  performed  to  examine  the  difference  between  these  two  categories.  A significant  difference  was  found 
(f=4.53,  px.OOl)  in  the  experimental  group,  looking  up  the  descriptive  data,  we  concluded  that  external 
motivation  was  higher  than  internal  motivation  for  the  experimental  group.  Facebook  tutoring  has  more 
effect  for  external  (environment)  motivation.  Table  3 displays  the  result  of  the  t-test. 


Table  3.  T-test  of  internal  and  external  motivation 


n 

Mean 

SD 

t 

P 

Internal  motivation 

30 

3.87 

.35 

4.53** 

.003 

External  motivation 

30 

4.11 

.72 

P<0.01** 
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Pearson  product  moment  correlation  coefficient  process  was  performed  to  examine  the  relationship 
between  learning  motivation  and  learning  achievement.  A significant  correlation  was  found  between  external 
(environmental)  motivation  and  learning  achievement  (r=.25,  p<.05).  However,  there  was  no  correlation 
between  internal  (personal)  motivation  and  learning  achievement.  Table  4 displays  the  result  of  the 
correlational  test. 


Table  4.  Correlational  test  for  learning  motivation  and  achievement 


n 

Pearson  r 

P 

Internal  motivation 

30 

,15 

.21 

External  motivation 

30 

.25* 

.04 

P<0.05* 

6.2  Facebook  Effects  Regarding  Student’s  Learning  Achievement 

According  to  the  instructional  design,  the  test  was  examining  student’s  achievement  on  vocabulary  and 
grammar.  First  of  all,  an  independent  t-test  was  performed  for  the  experimental  and  control  group  to  examine 
the  overall  effect  of  Facebook  tutoring,  that  is,  the  total  score  of  the  two  categories  (vocabulary  and  grammar) 
was  employed.  The  result  indicated  that  there  was  a significant  difference  between  the  two  groups  (f=2.56, 
p<.05)  in  terms  of  the  overall  learning  achievement.  This  result  indicated  that  Facebook  tutoring  had 
significant  effect  for  overall  learning  achievement.  Table  5 displays  the  result  of  the  t-test. 


Table  5.  T-test  of  learning  achievement 


n 

Mean 

SD 

t 

P 

Experimental  group 

30 

88.50 

7.32 

2.46* 

.017 

Control  group 

30 

83.83 

7.36 

P<0.05* 


We  further  looked  up  the  achievement  in  the  two  categories.  For  the  category  of  vocabulary,  significant 
result  was  also  found  in  the  independent  test  (t=2.91,  px.Ol).  For  the  category  of  grammar,  however,  the 
effect  of  Facebook  tutoring  was  not  significant  (t=.374,  p=.71).  This  finding  is  interesting.  We  believe  that 
vocabulary  is  easier  to  learn  and  remember  than  that  of  grammar,  due  to  the  fact  that  major  difference  in 
lingual  expressions  between  Chinese  and  English  does  exist.  It  is  hard  to  learn  English  times  for  most  of  the 
children  in  Taiwan.  The  specific  reasons  for  the  non-significant  of  the  grammar  learning  may  need  further 
investigations.  Table  6 and  table  7 display  the  result  of  the  t-tests. 


Table  6.  T-test  of  learning  achievement  on  vocabulary 


n 

Mean 

SD 

t 

P 

Experimental  group 

30 

36.25 

4.55 

2.91** 

.005 

Control  group 

30 

32.16 

6.18 

P<0.01* 

Table  7. 

T-test  of  learning  achievement  on  grammar 

n 

Mean 

SD 

t 

P 

Experimental  group 

30 

52.25 

6.44 

31 

.71 

Control  group 

30 

51.66 

5.62 
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6.3  How  did  Student  Learn  with  Facebook 

During  the  experiment,  the  instructor  conducted  a reflexive  log  in  which  his  observations  and  reflections  of 
student’s  learning  conditions  were  recorded.  We  summarized  the  important  notions  in  this  reflexive  log  in  the 
following  sections. 

6.3.1  Learning  Motivation 

We  found  that  there  was  a gain  in  learning  motivation  for  the  experimental  group.  With  compare  to  the 
control  group,  the  students  learned  with  Facebook  exhibited  more  positive  attitude  and  performed  more  peer 
interactions.  Students  seemed  to  prefer  more  to  discuss  the  questions  on  Facebook  than  in  the  classroom. 
Some  of  the  “quiet”  students  in  the  class  demonstrated  high  interest  in  replying  questions  and  argue  with 
peers  on  Facebook.  We  believe  some  students  who  usually  hesitate  to  express  their  thought  face-to-face  may 
have  more  intention  to  communicate  with  others  in  a virtual  environment. 

6.3.2  Learning  Activities 

Multiple  activities  (communication,  collaboration,  resource  sharing)  were  designed  as  the  instructional 
strategies.  Students  in  either  groups  felt  such  activities  were  interesting.  The  interactions  among  peers  were 
tremendously  increased.  We  believe  that  both  groups  were  benefitted  from  this  multiple  learning  activities, 
especially  for  the  experimental  group,  students  experienced  learning  in  a virtual  environment.  The  Facebook 
provided  a more  private,  more  flexible,  and  more  adaptive  environment  that  made  the  students  gain  better 
achievement. 

6.3.3  Learning  Style 

Student’s  learning  style  was  changed  for  both  groups.  Students  needed  to  prepare  what  to  discuss  and  how  to 
answer  the  questions  raised  either  by  teachers  or  peers.  Students  felt  they  were  participants  during  the 
learning  process  rather  than  listeners.  For  the  experimental  group,  student’s  messages  posted  on  Facebook 
could  be  seem  simultaneously  by  all  class  members.  Therefore  they  needed  to  be  more  careful  to  avoid 
possible  mistakes.  They  also  experienced  how  to  review  instructional  information  more  cautiously. 


7.  CONCLUSION 

This  study  intended  to  explore  the  effect  of  Facebook  tutoring  for  learning  English  as  a second  language.  The 
children  participated  the  Facebook  tutoring  generally  exhibited  a positive  attitude  toward  this  new  type  of 
learning  experience.  The  learning  achievement  for  the  Facebook  group  was  also  significantly  higher  than  the 
control  group. 

Specific  findings  regarding  learning  attitude  suggest  that  for  Facebook  tutoring,  student’s  external 
(environmental)  motivation  was  higher  than  internal  (personal)  motivation.  Further  exploration  on  the 
relationship  between  learning  motivation  and  learning  achievement  also  showed  that  external  motivation  was 
highly  correlated  with  the  learning  achievement.  These  phenomena  may  represent  that  Facebook  did  provide 
a flexible  environment  for  students  to  communicate,  collaborate,  and  share.  Positive  results  were  also  found 
in  learning  achievement.  There  was  a significant  difference  for  learning  achievement  between  Facebook 
tutoring  and  traditional  classroom  teaching.  However,  there  was  no  difference  on  grammatical  achievement. 
We  suspected  that  with  compare  to  Chinese,  English  grammar  has  major  differences  in  lingual  expressions, 
further  investigations  are  necessary  to  develop  proper  strategies  that  may  help  learn  English  grammar  on 
Eacebook. 

In  summary,  Eacebook  provides  the  unique  functions  that  facilitate  teachers  to  develop  activities 
incorporating  communicating,  collaborating,  and  sharing  strategies.  These  strategies  are  considered  to  be 
effective  in  learning  with  social  network.  This  study  evidenced  that  providing  with  proper  strategies,  both 
student’s  learning  attitude  and  learning  achievement  were  positively  affected  by  the  Eacebook  tutoring. 
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ABSTRACT 

This  work  focuses  on  an  experience  of  social  learning  realized  in  six  high  schools  in  Bari  (a  city  in  southern  Italy)  in  the 
2012-2013  Academic  year.  In  this  experience  we  used  ThinkTag  Smart,  a new  learning  platform,  to  train  400  students. 
After  an  introduction  concerning  Information  and  Communication  Technologies  in  the  Italian  school,  this  contribution 
will  describe  the  ThinkTag  Smart  platform  and  will  also  focus  on  the  experience  in  Bari,  underlining  special  aspects 
connected  to  social  construction  of  knowledge. 
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1.  INTRODUCTION 

One  of  the  most  prominent  features  of  contemporary  schools  is  that  students  are  born  and  raised  within  a 
context  which  takes  the  existence  of  Information  and  Communication  Technologies  (ICTs)  and  the  Internet 
for  granted  and  considers  them  as  “natural”.  They  have  been  diversely  labeled:  iGeneration,  Generation  Y, 
native  speakers,  digital  natives  (the  opposite  to  digital  immigrants).  Generation  Me  (Twenge,  2006), 
Millennial  Generation  (Howe  and  Strauss,  2003)  and  Net  Generation  or  Net  Gens  (Oblinger  and  Oblinger, 
2005).  Regardless  of  the  name,  this  generation  is  composed  of  students  born  between  approximately  1980 
through  2000  (Sutherland  and  Thompson,  2001;  Howe  and  Strauss,  2003).  As  Diana  and  Jim  Oblinger  (2005) 
point  out,  “as  long  as  they've  been  alive,  the  world  has  been  a connected  place,  and  more  than  any  preceding 
generation  they  have  seized  on  the  potential  of  networked  media”. 

In  Italy  despite  the  programs  implemented  by  the  Ministry  of  Education,  ICTs  are  coming  very  slowly 
into  the  school.  Italy  is  clearly  a "two  speed"  country:  from  one  side  widespread  computer  illiteracy  and  on 
the  other  side  a country  with  the  second  place  in  Europe  for  the  possession  of  mobile  phones,  with  children, 
the  Net  Gens,  who  govern  and  control  the  technological  tools  better  than  their  parents  and  teachers,  who  are 
not  always  able  to  exercise  adequate  control  for  their  proper  use  (Movimento  Difesa  del  Cittadino,  2007; 
Fieri,  2008;  Fieri  and  Diamantini,  2010)  or  to  use  them  in  the  learning  process  (Fieri,  2012).  In  the  national 
study  on  Digital  Schooling  (Avvisati  et  ak,  2013)  presented  to  the  Ministry  of  Education  in  Rome  it  clearly 
emerges  that  regarding  the  use  of  technology  in  schools  we  are  “late  compared  to  most  other  countries:  in 
2011  only  30%  of  Italian  students  in  their  third  year  of  middle  school  used  ICTs  as  a learning  tool  during 
science  lessons,  compared  to  an  average  of  48%  in  other  countries  of  the  OCSE”.  In  elementary  schools  on 
average  there  is  one  computer  for  every  15  children:  approximately  one  per  class.  There  is  one  FC  for  every 
1 1 students  in  middle  school  and  one  for  every  8 students  in  high  school.  One  classroom  out  of  five  in  Italian 
schools  has  an  interactive  multimedia  blackboard,  Lim.  Eight  schools  out  of  ten  have  internet  connections, 
but  only  have  of  the  classrooms  have  access  to  the  web.  According  to  OCSE,  the  Italian  project  for  the 
Digital  School  which  was  launched  by  the  MIUR  (Ministry  of  Education,  University  and  Research)  in  2007 
had  too  few  resources  “that  limited  the  effectiveness  of  its  initiatives.  Above  all,  the  report  shows  that  the 
lack  of  resources,  more  than  a lack  of  interest  on  the  part  of  schools  and  teachers,  is  the  reason  why  the 
presence  of  technology  is  still  very  low  in  the  classroom.”  OCSE  reports  that  for  multimedia  and  ICT  “30 
million  Euro  per  year  for  four  years  was  allocated  for  the  project  or  less  than  0.1%  of  public  expenditure  for 
education,  which  is  less  than  5 Euros  a year  per  student  in  primary  and  secondary  schools”. 
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In  this  contribution  we  will  present  an  experience  of  social  learning,  as  the  last  frontier  of  learning 
supported  by  ICTs,  in  an  Italian  high  school  realized  in  the  2012/2013  academic  year.  With  the  term  e- 
learning  2.0  we  refer  to  a type  of  computer-supported  collaborative  learning  (CSCL)  (Stahl  and  Hesse,  2009) 
system  that  developed  with  the  advent  of  the  Web  2.0  (Kerrer,  2006).  From  an  e-learning  2.0  perspective, 
conventional  e-learning  systems  were  based  on  instructional  packets,  which  were  delivered  to  students  using 
assignments.  Assignments  were  evaluated  by  the  teacher.  In  contrast,  the  new  e-learning  places  increased 
emphasis  on  social  learning  and  the  use  of  social  software  such  as  blogs,  wikis,  podcasts  and  virtual  worlds 
(Redecker,  2009).  Social  learning,  assumes  that  knowledge  (as  meaning  and  understanding)  is  socially 
constructed.  Learning  takes  place  through  conversations  about  content  and  grounded  interaction  about 
problems  and  actions  (Brown  and  Adler,  2008).  In  addition  to  virtual  classroom  environments,  social 
networks  have  become  an  important  part  of  e-leaming  2.0.  As  underlined  by  Sie  (2012)  “e-Learning  2.0,  is 
social  learning  through  and  through”.  One  of  the  most  important  features  of  e-learning  2.0  is  social  learning, 
which  assumes  that  the  best  way  to  learn  something  is  to  teach  it  to  the  others  (Bonfils,  Fieri  and  Dumas, 
2010;  Fieri,  Diamantini  and  Bonfils,  2011).  The  prosperity  of  social  networking  sites  has  met  this  demand. 
Open  source  social  networking  software  has  made  it  possible  to  create  social  networking  sites  with  only  a 
little  effort  (Ben  and  Chi,  201 1). 


2.  THINKTAG  SMART:  A SOCIAL  LEARNING  TOOL  AND 
METHODOLOGY 

ThinkTag  Smart,  the  learning  platform  focus  of  this  paper,  was  designed  to  aggregate  the  power  of  Web  2.0 
with  those  of  the  social  networks. 

The  ThinkTag  Smart  platform,  created  in  2005  by  the  Coaling  enterprise,  is  a quick  and  simple  way  to 
archive,  organize,  link  and  discuss  information  such  as  websites,  articles,  documents,  books,  movies  and 
music.  This  platform  mixes  the  learning  opportunities  offered  by  the  Web  2.0  with  the  learning  opportunities 
offered  by  social  networks.  This  platform  is  a tool  to  support  users  to  combine  the  requirements  of 
knowledge  management  and  the  opportunities  of  social  networks  to  share,  build  and  spread  knowledge  and 
information.  ThinkTag  Smart  is  a place  for  sharing  knowledge,  organizing  contents,  comparing  different 
kinds  of  knowledge  and  supporting  collaborative  work. 

The  main  characteristic  of  ThinkTag  Smart  is  the  “hyperpertinence”,  different  from  the  “ipertinenee” 
coined  by  sociologist  Derrick  de  Kerckhove  to  signify  the  particular  property  of  the  search  engines  of  the 
latest  generation  to  be  able  to  return,  in  the  face  of  a question  (query),  multiple  answers  that  pertain  to  it,  at 
different  levels,  both  of  language  (i.e.  texts,  images,  videos)  and  content,  in  short,  whatever  has  been 
uploaded  to  the  network  and  entertains  a relationship  with  that  query.  The  hyperpertinence  makes  this 
platform  different  from  social  networks  and  search  engines.  The  main  social  networks  in  fact,  focus  on  the 
relationship  between  people,  do  not  have  systems  for  structuring  and  organizing  contents  with  one  or  a few 
materials,  thus  they  are  irrelevant  with  respect  to  our  needs  for  managing  information.  The  search  engines  are 
able  to  reconstruct  the  networks  of  citations,  but  they  are  nothing  more  than  the  result  of  mere  algorithmic 
processing.  In  fact,  the  results  we  get  from  search  engines  like  Google  are  the  result  of  an  algorithm  that  can 
attribute  more  importance  to  the  more  "linked"  pages,  which  in  this  way  acquire  a high  ranking. 

The  ThinkTag  Smart  platform  is  able  to  aggregate  the  richness  of  the  network  with  the  social  construction 
of  meaning.  This  is  possible  because  each  resource  present  on  the  platform  must  be  necessarily  accompanied 
by  at  least  a title  and  one  or  more  tags.  The  network  environment  is  in  fact  “hyper”,  meaning  that  the  amount 
of  resources  that  can  be  identified  through  it  is  beyond  our  means  of  fruition/management.  At  the  same  time 
the  net  is  approximate,  uncoordinated  and  often  irrelevant  and  brings  to  an  "information  overload".  This 
indicates  that  the  more  information  you  have,  the  more  information  you  must  have  in  order  to  manage.  To 
avoid  “information  overload”  and  “confusion”,  every  resource  in  the  platform  must  have  a title  and  one  or 
more  “tags”.  These  tags  are  carried  by  all  the  users.  In  ThinkTag  Smart  the  construction  of  meaning  and  the 
organization  of  resources  is  self-determined  and  bottom  up. 

As  time  goes  by,  the  resource  cumulates  the  tags  of  all  the  users;  this  means  that  the  more  ideas 
exchanged  on  a subject  the  more  tags  we  will  have  on  that  subject.  This  artifact  is  used  by  other  social 
networks,  and  is  named  "tag  cloud".  The  tag  cloud  is  used  to  reconstruct  a visual  map  of  the  tags  used  by 
users  or  the  main  tags  used.  In  the  tag  cloud  each  tag  is  a link  to  the  resources  that  use  that  word  and  appears 
in  different  sizes  depending  on  the  frequency  of  use,  so  the  tags  displayed  in  large  characters  are  among  the 
ones  most  used  by  users. 
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In  addition  to  the  standard  tag  cloud,  ThinkTag  Smart  has  developed  an  innovative  feature  called 
“hyperpertinent  tag  cloud”  that  provides  users  with  a graphical  output  that  can  display  not  only  the 
occurrence  of  the  different  tags  but  also  the  relationships  between  the  different  tags.  Starting  from  an  initial 
tag  (identified  through  the  search  engine  or  by  clicking  on  a tag  of  a resource)  the  user  gets  a new  cloud  that 
shows  (with  different  sizes)  all  the  tags  used  by  the  users  along  with  the  initial  tag.  With  the  hyperpertinent 
tag  cloud,  it  is  possible  to  reconstruct  the  network  of  relationships  emerging  between  the  resources  reported 
by  users  on  a particular  subject. 

It  is  important  to  underline  that  the  tag  will  not  be  made  by  leaders  or  by  experts,  but  by  all  the  users.  The 
construction  of  meaning  and  the  organization  of  resources  are  then  from  below;  it  is  self-determined  and 
follows  the  bottom  up  logic.  This  is  certainly  a source  of  added  value,  in  that  it  allows  to  read  the  same 
resource  from  different  points  of  view;  in  this  way  knowledge  is  not  flat,  but  is  a prism. 

The  main  elements  of  ThinkTag  Smart  are: 

RESOURCES:  any  type  of  document  or  material  produced  by  users  or  derived  from  the 
productivity  of  others,  which  users  decide  to  share  (ThinkTag  Smart  can  handle  more  than  fifty 
different  types  of  resources). 

SHELVES:  containers  can  be  used  as  a virtual  file.  The  shelves  allow  to  bring  together  a range 
of  resources  that  are  considered  relevant  or  related  to  a unique  concept.  The  shelf  is  born  as 
individuaEpersonal  and  only  the  person  who  created  it  can  modify  it  and  add  content  to  it,  even 
if  everyone  can  see  it. 

COLLECTIONS:  follow  the  same  logic  of  the  shelf,  but  they  are  not  personal  but  of  the  group, 
all  members  of  the  group  can  add  and  edit  content. 

CONNECTIONS:  tie  a content  to  another  content  without  putting  them  in  the  same  "container". 
GROUPS:  group  of  people,  collaborative  community  with  a common  goal  to  work  on. 
Administrators  have  the  possibility  to  edit  the  group  settings,  manage  members  and  in  general  to 
change  several  aspects  of  the  group,  the  members  however,  participate  in  the  "life  of  the  group," 
but  without  having  the  possibility  to  handle  it. 

All  in  all,  we  can  compare  ThinkTag  Smart  to  a bottom-up  process,  where  the  initiatives,  the  materials, 
the  creation  of  groups  and  the  sharing  of  knowledge  are  derived  from  the  spontaneous  initiative  of  users 
surfing  the  web  following  their  own  personal  interests.  The  main  objective  of  this  approach  is  to  make 
explicit,  and  therefore  more  accessible,  the  tacit  and  implicit  knowledge  that  every  user  has.  The  users 
sharing  opinions  and  knowledge  enrich  their  own  personal  baggage,  automatically  helping  to  enrich  and 
update  the  knowledge  of  the  network  of  users  connected  to  them. 

Thanks  to  the  tags  and  the  connections  between  resources,  users  can  organize  the  knowledge  collected, 
interpreting,  refining  and  assembling  it  in  an  appropriate  way  for  re-use.  The  platform  instantly  processes  a 
map  that  graphically  depicts  the  connections  created  between  resources  related  to  the  same  topic,  in  order  to 
provide  an  overall  clear  and  immediate  picture  of  the  network.  The  set  of  resources,  with  their  labels,  in  this 
way  becomes  a dynamic  and  ever-changing  system  of  relationships  between  objects. 

With  the  use  of  tags  we  are  witnessing  a change  in  the  classification  of  documents:  the  "taxonomies"  (the 
so-called  "folksonomies"):  users  can  assign  semantic  labels  to  documents  to  facilitate  classification  and 
research.  To  assign  labels  users  must  put  themselves  in  the  shoes  of  others,  thinking  about  the  words  that 
other  people  would  use  to  find  the  resources  connected  to  the  specific  topic. 

Another  important  aspect  of  knowledge  management  is  the  ability  to  use  the  knowledge  generated  to 
create  new  knowledge.  The  premise  of  this  activity  is  the  internalization  of  exchanged  knowledge,  to  take 
possession  of  new  thought  patterns  that  are  applicable  to  daily  and  professional  life.  Even  wikis  support 
collaborative  learning  by  allowing  users  to  create  shared  documents  so  that  everyone  can  intervene  by 
modifying  and  adding  parts  quickly  and  easily.  In  conclusion,  this  kind  of  platform  offers  tools  and 
opportunities  for  the  kind  of  activities  (school,  work  and  personal)  which  require  a broadening  of 
perspectives  in  a social  perspective.  The  users,  by  sharing  and  comparing,  will  evolve  while  enriching 
themselves  (creating  new  knowledge)  and  others  (sharing). 

Regarding  the  safety  of  the  platform,  this  platform  is  dedicated  only  to  schools,  and  though  it  is  available 
on  the  internet  it  does  not  provide  access  to  users  who  are  not  part  of  the  project.  All  students  were  able  to 
have  only  one  account  corresponding  to  their  first  and  last  name.  Users  with  different  nicknames  were  not 
accepted,  when  different  nicknames  were  created  by  some  students  they  were  canceled  without  informing  the 
user.  Constant  monitoring  of  the  platform  identified  some  (few)  enrollments  made  by  external  users  (in  some 
cases  teachers  who  were  accepted,  in  other  cases  unknown  users  who  were  blocked/refused).  Apart  from  the 
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enrollments  to  the  platform,  the  main  issue  is  the  access  to  groups  and  the  possibility  to  add  resources  for  the 
groups.  The  task  of  managing  the  groups  was  entrusted  to  teachers  and  students  who  wanted  to  take  on  this 
responsibility.  The  author  of  the  resource  assumes  responsibility  for  what  he  publishes,  while  the 
administrators  of  the  group  in  addition  to  deciding  who  joins  the  group  can  also  decide  whether  resources  to 
be  published  should  be  subjected  to  a preliminary  assessment  within  the  group.  Following  the  issue  of 
crowdsourced  content  validation,  the  contents  developed  and  proposed  were  consistently  evaluated  by  the 
community.  The  platform  was  continuously  monitored,  and  there  were  no  cases  of  problematic  content.  Just  a 
few  cases  of  generally  offensive  comments  were  reported,  but  thanks  to  the  incisive  and  rapid  intervention  of 
some  users,  we  decided  not  to  remove  the  comments. 


3.  CASE  STUDY:  “SOCIAL  LEARNING  LABS  FOR  SMART  LEARNING 
COMMUNITIES” 

This  experience,  realized  in  the  2012/2013  academic  year  in  six  high  schools  of  Bari  (a  city  in  southern 
Italy),  involved  18  classes,  400  students  and  50  teachers.  In  December  2012  we  held  the  training  of  teachers 
(about  the  methodological  aspects  and  about  the  use  of  the  platform);  in  January  2013  we  started  the 
activities  with  the  students. 

The  project  applied  the  methodologies  and  instruments  of  the  web  2.0,  directly  involving  the  students  and 
teachers  in  the  creation,  search,  management  and  organization  of  suitable  contents  in  order  to  structure  18 
learning  areas  (i.e.  “The  theatre:  the  spaces,  the  words,  the  experiences  over  time”;  “The  revolutions  of  the 
1700s”;  “Adolescents  and  alcohol”). 

The  activities  began  in  January  2013  and  continued  until  the  end  of  April,  involving  teachers  and  students 
in  the  construction  of  contents  for  the  various  subjects  - literature,  social  sciences,  history,  geography,  art, 
math,  natural  science,  foreign  languages  and  literature  (English,  French,  Spanish  and  German),  artistic  and 
creative  arguments  - as  part  of  the  project  in  the  creation  of  54  groups  using  over  3,800  resources,  more  than 
800  shelves  and  100  collections,  with  countless  connections  among  resources  in  a constant  and  significant 
activity  of  interaction. 

During  the  learning  path  the  creation  of  resources  had  a constant  progression  over  time  with  the 
development  of  the  didactic  activity  (Figure  1).  Regarding  the  creation  of  the  shelves  there  is  a regular  trend, 
with  an  increment  corresponding  to  the  need  to  organize  contents  and  information  that  was  acquired  and  used 
in  the  project  during  the  last  three  weeks  (Figure  2).  Regarding  the  collections,  which  are  an  instrument  for 
organization  shared  inside  groups,  their  use  was  seen  after  the  first  month  of  work  and  constantly  grew, 
which  confirms  the  acquisition  of  a shared  and  active  work  style  inside  the  groups  (Figure  3).  Regarding  the 
connections,  which  represent  the  organization  of  the  contents  in  a network,  it  became  more  intense  in  the  last 
month  of  the  project  (Figure  4).  This  confirms  the  acquisition  of  the  value  of  this  tool  for  handling  the 
multiplicity  and  variety  of  the  contents.  Finally,  regarding  the  groups,  after  an  intense  development  in  the 
first  phase  (in  the  first  week)  to  create  project  communities,  the  situation  stabilized  with  the  creation  of  a few 
communities  focused  on  specific  topics  which  emerged  during  the  project  (Figure  5). 
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Figure  2.  Shelves 


Figure  3.  Collections 


Figure  4.  Number  of  connections 
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Groups 


Figure  5.  Groups 

A distinguishing  feature  of  the  learning  process  was  the  identification  of  a tutor  among  the  members  of 
the  “project  group”  of  the  community.  The  tutor  is  a student,  who  gives  advice  and  indicates  corrections  that 
need  to  be  made. 

The  specific  modality  given  by  the  platform,  which  is  the  real  method  of  social  learning,  has  helped  some 
students  to  overcome  difficulties  in  their  social  relationships  and  in  communications,  taking  on  a new  role 
with  the  possibility  to  construct  new  relationships  inside  the  group  they  belong  to. 

The  use  of  “comments”  was  transversal  and  involved  most  of  the  users: 

the  teachers  used  it  to  support  activities,  to  offer  suggestions,  work  strategies,  corrections  or 
areas  to  focus  on 

the  students  used  it  to  indicate  how  they  used  the  platform,  to  suggest  ideas  to  work  on  and  for 
their  opinions  on  contents. 

A distinguishing  and  particular  feature  Smart  platform  to  evaluate  contents  negatively  or  positively.  As 
can  be  seen  in  the  chart  “Reputation  score”,  55.1%  of  the  participants  significantly  increased  their  reputation 
(from  10  to  100  or  more  points,  specifically  34%  of  the  experience  was  the  use  that  was  made  of  the 
possibility  given  by  the  ThinkTag  Smart  with  more  than  200  points)  (Figure  6). 
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4.  CONCLUSIONS 

In  this  experience  students  were  guided  to:  actively  participate  in  the  different  communities;  not  search  only 
for  learning  models  that  were  delivered;  construct  together  the  course  contents  and  materials  based  on  the 
lessons  and  follow  up  activities;  organize  their  own  learning  path;  participate  in  the  shared  assessment  of  the 
contents  and  materials;  organize  their  own  materials  in  a useful  way  during  the  whole  learning  process; 
identify  their  approach  to  the  complexity  of  the  knowledge  and  of  the  learning.  The  project  obtained  the 
following  results:  activation  and  involvement  of  the  students  in  the  “construction”  of  their  learning  path, 
availability  of  an  explicit  body  of  knowledge  to  use  in  the  different  courses  and  availability  of  a network  of 
knowledge  for  the  development  of  interdisciplinary  approaches  to  educational  fields. 

As  reported  by  a student  “When  I enrolled  on  this  platform,  honestly  I was  not  very  enthusiastic,  but  then 
during  the  course,  learning  new  things,  being  together  with  friends  and  teachers,  I liked  it  a lot.  I believe  that 
by  working  on  the  computer,  using  the  technology  we  learn  better,  and  it  is  much  more  fun  and  easier.  The 
great  thing  about  this  platform  is  that  you  can  also  add  different  resources,  such  as:  images,  texts,  pictures 
etc.  ...  I mean,  maybe  even  in  the  future  we  will  use  this  method,  because  it's  really  helpful”.  Another  student 
underlines  that  “This  platform  proved  to  be  especially  useful  for  my  work,  my  reports,  my  comments,  and  my 
careful  selection  of  resources  on  the  network,  none  of  this  has  remained  confined  within  the  four  walls  of  my 
classroom,  but  in  fact  it  was  appreciated  and  judged  by  other  people,  so  that  maybe  my  resources  have 
proved  useful  for  the  study  of  some  of  the  content  or  for  a more  in-depth  review  of  the  assignment.  In  a 
certain  sense,  through  ThinkTag  Smart  I also  able  to  express  myself  more”.  In  conclusion,  with  the  words  of 
one  of  the  teachers  who  participated  in  the  experience,  «a  student,  who  really  worked  hard  said:  “Professor, 
I have  worked  on  many  projects  in  this  school:  I took  a course  on  Public  Law  at  the  Faculty  of  Law  in  Bari 
with  a final  exam,  I went  to  Rome  to  present  a project  to  the  BCE,  but  this  project  has  been  the  most  difficult 
one.”  I asked  him  to  explain  why  and  he  answered:  “it  was  very  complex;  there  were  too  many  connections 
to  remember  which  then  needed  to  be  clarified  and  then  carried  over  into  our  work  in  a simpler  format  in 
order  to  make  them  clear  for  the  students  using  the  e-book.  It  was  a real  brain-teaser.”  I asked  him:  “But  in 
the  end  you  did  it,  right?”  He  answered  simply:  “Yes.”  “Then”,  I told  him,  “you  reached  the  objectives  that 
this  activity  aimed  to  have  you  reach:  learn  how  to  manage  the  complexity  of  the  knowledge  of  the  overall 
picture  and  to  see  the  connections  in  order  to  make  critically  aware  choices.  Outside  school  there  isn’t  the 
reality  of  math,  business  economics,  or  law.  There  is  reality  and  stop,  knowing  how  to  discern  the  various 
aspects  and  how  to  work  in  a relevant  way  is  a competence  which  is  acquired  with  hard  work  that  you 
yourself  experienced.”  Your  smile  of  satisfaction  that  you  gave  us  coming  out  of  the  lab  made  up  for  the 
immense  work  that  we  did  all  together  in  order  to  reach  this  small  result,  a drop  in  an  immense  sea  of  things 
to  do  in  school.». 

In  the  Italian  context,  and  not  only,  experiences  of  this  kind  can  contribute  in  an  important  way  to: 

sustain  the  switch  to  digital  in  schools,  trying  to  bridge  the  digital  divide  which  is  still  strong 
among  teachers  and  students,  the  Net  Gens 
update  schools  to  the  ongoing  transformations 

support  the  “educational  reconversion”  of  the  new  social  media  and  social  networking 
reconnect  the  break  between  the  cognitive  experience  of  students  and  the  school 
overcome  the  current  logic  of  schools 

directly  involve  students  in  the  creation,  management  and  organization  of  contents  to  create 
greater  awareness  and  administration  of  what  is  learned 
give  value  to  interaction  and  collaboration 

give  methodologies  and  instruments  to  administer  informal  learning  integrated  with  formal 
learning 

produce  digital  contents  using  a bottom  up  process. 
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ABSTRACT 

Distance  education  has  become  a rather  popular  form  of  education  recently.  The  advantages  of  this  form  are  obvious  and 
well-known.  They  include  asynchronous  learning,  individualized  learning  trajectories  and  convenient  case  technologies. 
However,  the  distance  form  of  education  is  not  able  to  form  the  trainee’s  hands-on  experience,  especially  concerning  the 
complicated  technological  equipment.  Besides,  it  cannot  form  one’s  personal  communication  skills  of  working  in  the 
team.  That’s  why  many  distance  teaching  programs  include  synchronous  teaching  elements.  The  difference  in  the 
students’  achievements  creates  a lot  of  problems  in  the  organization  of  educational  groups.  The  principle  of  dynamic 
study  group  forming  for  a short-term  synchronous  teaching  is  proposed  by  the  author  as  a way  of  solving  such  problems. 


KEYWORDS 

Distance  learning,  individualized  educational  trajectory,  educational  process  control. 


1.  INTRODUCTION 

The  development  of  modern  e-learning  platforms  (Cecilia  Rossignoli  et  al,  2013;  Margot  McNeill  et  al, 
2010)  and  electronic  search  and  processing  marked  the  highest  priority  of  distance  learning.  The  problem  of 
new  effective  pedagogical  technologies  development  has  been  paid  a lot  of  attention  to  recently.  Researchers 
(Dr.  Ajay  Kumar  Attri,  2012;  Said  Hadjerrouit,  2013)  propose  methods,  strategies  and  technologies  of  the 
distance  learning.  Further  research  into  course  development  techniques  will  help  educational  institutions 
understand  which  methods  work  best  in  the  distance  teaching. 


2.  DISTANCE  EDUCATION:  EDUCATIONAL  TRAJECTORY 
INDIVIDUALIZATION 

Extra-mural  education  is  the  closest  in  its  form  to  distance  education  in  the  Russian  higher  education  system. 
The  peculiarity  of  this  education  form  is  the  integration  of  asynchronous  teaching  and  synchronous  final 
control.  The  curriculum  is  formed  in  the  following  way  (Fig.  1):  the  first  one  or  two  weeks  are  dedicated  to 
the  students’  starting  acquaintance  with  the  curriculum  contents,  the  study,  as  a rule,  being  organized  in  the 
mode  of  synchronous  teaching.  The  future  student’s  training  is  realized  in  asynchronous  teaching  form  and 
presupposes  the  student’s  individual  work  with  the  published  sources,  recommended  Internet  resources  and 
the  information  of  the  case  provided.  At  the  end  of  the  course  the  student  is  to  fulfill  practical  tasks  and 
laboratory  practice  in  synchronous  teaching  mode.  The  course  is  completed  with  final  examination  (Titova 
O.V.  and  Kravets  A.G.,  2013). 

Such  organization  of  educational  process  is  realized  in  Eurasian  Open  Institute,  London  School  of 
Business  and  Finance  (LSBF),  National  University  (USA),  National  University  - UNED  (Spain)  and  other 
systems  of  open  education.  Nevertheless,  it  should  be  taken  into  consideration  that  the  efficiency  of  students’ 
achievements  in  the  distance  teaching  mode  depends  on  their  self -organizational  ability  and  effective  system 
of  inner  motivation  to  a great  extent  [ 1 ] . 
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Figure  1.  Time  portrait  (initial)  of  the  distance  education  process  structure 

According  to  the  inertial  subject  concept,  proposed  by  us,  the  process  of  mastering  the  curriculum  by  the 
student  is  nonlinear  (Fig.  2)  and  consists  of  the  parts  represented  one  after  another  [2,  3]: 

AB  - student’s  interest  development  (mastering  the  means  used  in  the  particular  field  of  knowledge); 

BC  - information  component  widening  (the  set  of  tasks  solved  with  the  mastered  means); 

CD  - elaborating  the  techniques  and  methods  of  specialized  information  processing; 

DE  - optimizing  the  ways  of  information  storage  (knowledge  transformation  onto  personal  level); 

EF  - new  specialized  means  development. 

On  the  diagram  (Fig.  2)  the  value  of  Z characterizes  the  student’s  real  achievements  level,  quantitatively 
determined  as 

i=l 

where  k;  is  the  vector  of  the  weight  coefficients  determining  the  significance  of  this  or  that  competence  in 
the  professional  portrait  of  the  graduate,  is  the  rating  grades  vector  determining  the  level  of  mastering  the 
competence. 


Figure  2.  Individualized  educational  trajectory 

In  a number  of  works  we  substantiate  that  the  necessary  condition  of  forming  the  student’s  stable 
motivation  is  creating  the  range  of  acceptable  deviations  Az  of  one’s  real  educational  trajectory  (Fig.  2 line  a) 
from  the  defined  one  (Fig.  2 curve  b).  Az  range  narrowing  to  the  level  lower  than  acceptable  one  results  in 
student’s  overestimated  self-appraisal  forming  and,  therefore,  lowering  the  level  of  further  studies 
motivation.  Vice  versa,  Az  range  widening  forms  the  feeling  of  apathy  because  the  student  understands  his  or 
her  inability  to  achieve  the  result. 

On  the  basis  of  these  developments  the  distance  education  management  system  was  created  by  the  author, 
its  pilot  project  being  realized  in  Svetly  Yar  branch  of  Moscow  State  University  of  Technologies  and 
Management  named  after  K.G.  Razumovsky.  Its  main  characteristic  feature  is  distributing  the  time  resource 
for  synchronous  teaching  realization  inside  the  interval  of  students’  distance  consulting.  As  the  educational 
process  is  inertial,  this  distribution  is  nonlinear:  the  greatest  intensity  of  synchronous  education  periods  is 
displaced  to  the  beginning  of  teaching.  Such  educational  process  structure  allows  the  students  to  master  the 
means  used  in  the  educational  course  in  the  most  effective  way.  These  means  include  terminology  (concepts 
and  definitions),  methods  and  algorithms  of  data  processing  and  work  with  specialized  IT  contents. 
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The  following  algorithm  serves  as  the  basis  of  this  system: 

1.  Developing  the  individualized  task  according  to  the  curriculum.  The  task  consists  of  a number  of 
smaller  tasks  to  be  solved  by  the  student  one  after  another,  the  results  being  the  source  data  for  the  final  task 
according  to  the  curriculum. 


Initial  training  Distance  consultation  Exams 

module 


Figure  3.  Time  portrait  of  the  distance  education  process  structure  proposed 

2.  The  whole  time  interval  that  the  student  has  for  individual  work  is  divided  into  small  time  parts  with 
control  reports.  Such  reports  are  carried  out  either  in  the  off-line/on-line  mode  of  testing  or  on-line  interview. 

3.  According  to  testing  results  it  is  decided  if  additional  consultations  are  necessary,  the  subject,  volume 
and  form  being  taken  into  consideration. 

4.  Study  groups  are  formed  for  short-term  synchronous  teaching.  The  aim  of  this  teaching  is  developing 
students’  professional  competences  necessary  when  operating  real  equipment. 

5.  The  organization  of  students’  final  control. 

It  should  be  stressed  that  the  study  groups’  structure  was  not  strictly  defined  and  the  groups  were  formed 
according  to  the  intermediate  control  of  all  the  students  in  parallel  groups. 


3.  TESTING  THE  PROPOSED  EDUCATIONAL  PROCESS  STRUCTURE 

To  check  the  efficiency  of  the  proposed  teaching  process  two  groups  were  formed:  the  students’  experimental 
group  A (36  people)  and  control  group  B (38  people).  The  groups  consisted  of  the  extra-mural  students  in 
Svetly  Yar  branch  of  Moscow  State  University  of  Technologies  and  Management  named  after  K.G. 
Razumovsky. 

The  analysis  of  group  partieipants’  competence  level  is  represented  in  two  stages,  fixing  and 
transformational  ones.  On  the  first  fixing  stage  the  comparability  analysis  of  two  students’  groups  (A  and  B) 
was  carried  out  as  for  the  difference  of  their  starting  level  achievements.  According  to  the  results  of  the 
starting  testing,  students  were  divided  into  three  subgroups  characterized  by  low,  intermediate  and  advanced 
levels  of  students’  starting  achievements. 
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Table  1.  Groups  qualitative  analysis  according  to  academic  achievements  (according  to  the  starting  control 

results) 


Assessment  criterion 

No. 

Number 

Formula 

Low 

level 

Intermediate 

level 

Advanced 

level 

For  group  A (experimental) 


1 

Students  total,  people 

N 

36 

2 

Absolute  number,  people 

Na 

20 

11 

5 

3 

Relative  number,  % 

5 

N 

= ^•100% 
N 

55.6 

30.6 

13.9 

For  group  B (control, 

) 

1 

Students  total,  people 

N 

38 

2 

Absolute  number,  people 

Ns 

25 

10 

3 

3 

Relative  number,  % 

8 

o 

o 

II 

65.8 

26.3 

7.9 

Figure  4.  Teaching  process  algorithm  in  distance  education 

Let  us  establish  the  characteristics  matching  or  difference  in  the  experimental  and  control  groups.  For  this 
purpose  let  us  propose  statistical  hypotheses: 

- the  hypothesis  of  differences  absence  (a  null  hypothesis); 

- the  hypothesis  of  differences  significance  (an  alternative  hypothesis). 
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Statistical  tests  are  used  to  decide  which  hypothesis  should  be  accepted.  In  our  research  Welch’s  f-test  is 
used  for  testing  the  hypothesis  concerning  the  characteristics  matching  in  two  groups  [4].  The  value,  which  is 
called  the  observed  test  value,  is  calculated  on  the  basis  of  observation  results.  This  value  is  compared  with 
the  known  reference  value  specified  in  the  tables,  which  is  called  the  critical  test  value.  In  pedagogical 
researches  the  value  of  0.05  is  usually  the  limit  that  means  the  possibility  of  an  acceptable  mistake  being  less 
than  5%. 

If  the  observed  test  value  obtained  proves  to  be  less  than  the  critical  one  or  equal  to  it,  the  null  hypothesis 
is  accepted;  the  control  and  experimental  groups’  characteristics  match.  Alternatively,  if  the  observed  test 
value  proves  to  be  strictly  higher  than  the  critical  one,  the  null  hypothesis  is  declined  and  the  alternative 
hypothesis  is  accepted:  experimental  and  control  groups’  characteristics  are  considered  different,  the 
difference  validity  being  0.95  or  95%. 

The  observed  value  of  Welch’s  f-test  is  calculated  according  to  the  following  formula 

-N  \x- y\  

T = , , where  N,  M - sample  sizes  of  x and  y,  x,  y - sample  means,  D , D - sample 

-D^+N-D^,  ' " 

variances. 

To  calculate  the  main  characteristics  the  following  formulae  were  used: 

x(y)  = — {k.x.  + -I- . . .-I-  kf,Xf, ) , where  k,  - weight  significance  coefficients, 

N 

Dm)  = ^—y(x,-xf . 

Students’  experimental  group  A 

Sample  mean:  x = — (20  • 1 -I- 1 1 • 2 -I-  5 • 3)  = 1.58  ■ 

36 

Sample  variance:  [20(1 -1.58)" -fl  1(2-1.58)" -f  5(3-1.58)" ] = 0.54 • 

36-1 


Students’  control  group  B 

Sample  mean:  y = — (25-1-f  10- 2-f  3-3)  = 1.42- 

38 


Sample  variance;  [25(1  - 1.58)"  -f- 10(2  - 1.58)"  +3(3-1.58)"]  = 0.41. 

38  — 1 


The  observed  value  of  Welch’s  f-test:  f = 


V36-38I1 .58-1.421 


= 1.02- 


V36- 0.54-^38-0.41 

Hence  r = 1.02  <1.96.  Therefore,  the  hypothesis  of  control  and  experimental  groups’  characteristics 
matching  before  the  beginning  of  the  experiment  is  accepted  on  the  significance  level  of  95%. 


4.  THE  EXPERIMENT  RESULTS  EVALUATION 

The  educational  process  for  control  group  B did  not  undergo  any  changes  either  in  the  teaching  course 
structure  or  in  the  principle  of  study  groups’  forming. 

For  group  A the  time  resource  at  the  end  of  the  educational  course  for  synchronous  teaching  realization 
was  divided  into  smaller  time  intervals  that  were  evenly  distributed  in  the  time  range  for  individual  work. 
Subgroups  of  group  A students  were  formed  for  carrying  out  short-term  teaching  in  each  such  interval.  The 
main  choice  criterion  was  the  student’s  achievements  fixed  level  that  was  evaluated  on  the  basis  of  current 
control  in  the  form  of  on-line  testing. 

The  following  results  (Table  2)  were  obtained  at  the  final  examination  of  both  groups’  students. 
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Table  2.  Groups  qualitative  analysis  according  to  academic  achievements  (according  to  the  final  control  results) 


Assessment  criterion 

No. 

Number 

Formula 

Low 

level 

Intermediate 

level 

Advanced 

level 

For  group  A (experimental) 


1 

Students  total,  people 

N 

36 

2 

Absolute  number,  people 

Aa 

9 

21 

6 

3 

Relative  number,  % 

N 

5 = ^-100% 
N 

25.0 

58.3 

16.7 

For  group  B (control) 


1 

Students  total,  people 

N 

38 

2 

Absolute  number,  people 

Nb 

21 

14 

3 

3 

Relative  number,  % 

N 

5 = ^-100% 
N 

55.3 

36.8 

7.9 

To  evaluate  the  comparability  of  the  two  study  groups,  control  and  experimental  ones,  the  above 
mentioned  method  was  used.  The  obtained  Welch’s  f-test  value  was  T = 2.59. 

As  T = 2.59  >1.96,  the  hypothesis  concerning  the  difference  in  characteristics  of  control  and 
experimental  groups  after  the  teaching  process  is  true  on  the  significance  level  of  95%. 

The  qualitative  structure  of  experimental  group  A and  control  group  B according  to  the  starting  and  final 
testing  is  represented  graphically  in  Figures  5 and  6. 


Low  Intermediate  Advanced 


achievements  achievements  achievements 
level  (1)  level  (2)  level  (3) 

Figure  5.  Comparative  diagram  of  the  starting  testing  results  for  experimental  (A)  and  control  (B)  groups 


level  (1)  level  (2)  level  (3) 


Figure  6.  Comparative  diagram  of  the  final  testing  results  for  experimental  (A)  and  control  (S)  groups 
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According  to  the  academic  achievements,  the  qualitative  structure  of  experimental  group  (A)  has  changed; 
the  number  of  students  with  intermediate  and  advanced  achievements  level  (Table  3)  has  increased.  Welch’s 
f-test  value  was  3.5  when  academic  achievements  level  in  group  A before  and  after  teaching  were  compared. 
It  proves  substantial  differences  in  the  academic  achievements  levels  in  this  group  in  the  beginning  and  at  the 
end  of  teaching. 


Table  3.  Experimental  group  A qualitative  structure  analysis  according  to  the  academic  achievements  (in  the 

starting  and  final  testing) 


Assessment  criterion 

No. 

Number 

Formula 

Low 

level 

Intermediate 

level 

Advanced 

level 

Starting  testing 


1 

Absolute  number 

(students  total:  36),  people 

Na 

20 

11 

5 

2 

Relative  number,  % 

N 

5 = ^-100% 
N 

55.6 

30.6 

13.9 

Final  testing 


1 

Absolute  number,  people 

Na 

9 

21 

6 

2 

Relative  number,  % 

N 

5 = ^-100% 

25.0 

58.3 

16.7 

N 

Relative  number  change: 

30.6  4. 

27.7  t 

2.8  t 

Some  changes  of  group  qualitative  structure  according  to  the  academic  achievements  level  (Table  4)  have 
also  taken  place  in  control  group  (B).  Nevertheless,  Welch’s  f-test  value  compared  before  and  after  teaching 
in  this  group  was  0.7  (<1.96).  It  proves  the  fact  of  the  group  qualitative  structure  to  be  unchanged  up  to  95%. 


Table  4.  Control  group  B qualitative  structure  analysis  according  to  the  academic  achievements  (in  the  starting 

and  final  testing) 


Assessment  criterion 

No. 

Number 

Formula 

Low 

level 

Intermediate 

level 

Advanced 

level 

Starting  testing 


1 

Absolute  number 

(students  total:  38),  people 

Nb 

25 

10 

3 

2 

Relative  number,  % 

5 = i^-100% 
N 

65.8 

26.3 

7.9 

Final  testing 


1 

Absolute  number,  people 

Nb 

21 

14 

3 

2 

Relative  number,  % 

5 = i^-100% 

55.3 

36.8 

7.9 

N 

Relative  number  change: 

10.5  4. 

10.5  t 

0 

5.  CONCLUSIONS 

In  the  pilot  project  of  managing  the  extra-mural  students’  educational  process  in  Svetly  Yar  branch  of 
Moscow  State  University  of  Technologies  and  Management  named  after  K.G.  Razumovsky  the  following 
work  was  carried  out: 
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- starting  testing  technology  to  determine  real  level  of  students’  achievements  was  developed  and 
introduced; 

- intermediate  control  technology  for  the  current  level  of  students’  achievements  was  developed  and 
introduced; 

- the  main  principles  of  reorganizing  the  teaching  course  to  actualize  the  motivation  system  taking  into 
consideration  the  students’  individual  characteristics  were  developed; 

- the  main  principles  of  the  teaching  course  to  allow  integrating  the  advantages  of  synchronous 
education  into  the  extra-mural  form  according  to  the  students’  individual  characteristics  were  developed. 

The  following  results  were  obtained: 

- objective  assessment: 

- changing  the  qualitative  structure  of  the  study  group  according  to  the  academic  achievements 
(Table  3):  the  number  of  students  with  low  academic  achievements  level  has  decreased; 

- the  level  of  general  achievements  in  experimental  group  A is  higher  than  in  control  group  B; 

- subjective  assessment: 

- students’  interest  to  their  learning  has  increased; 

- students’  perception  level  of  the  curriculum  has  increased; 

- positive  basis  for  educational  process  perception  by  the  students  has  been  formed. 
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ABSTRACT 

This  paper  is  a case  study  of  a vocational  high  school  in  Taiwan.  The  main  purpose  of  the  present  study  is  to  investigate 
the  key  determinants  of  a school’s  success  in  initiating  and  sustaining  pedagogical  innovations  in  Information  and 
Communications  Technology  (ICT)  implementation,  with  a specific  focus  on  the  effect  of  leadership  approaches  in  the 
change  process.  Centring  on  a particular  vocational  high  school,  this  study  examined  the  way  in  which  the  entire  staff 
succeeded  in  transforming  a traditional  school  into  an  acknowledged  ICT-capable  school.  Questionnaires  and  interviews 
served  as  the  main  research  methods  within  this  study.  The  results  confirmed  that,  first,  collaborative  leadership  was  at 
the  core  of  successfully  managing  changes  of  ICT  integration.  Second,  the  findings  highlight  the  importance  of  fostering 
the  future  leaders  for  sustaining  good  practices  of  ICT  implementation.  Finally,  the  findings  reflect  upon  the  potential 
impact  of  the  leadership  approaches  on  teachers’  engagement  in  the  change  process  of  ICT  implementation.  Currently, 
there  is  still  limited  research  focusing  on  schools’  sustainability  of  ICT  implementation.  The  present  research  could  serve 
as  a reference  for  further  research  in  this  regard. 
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1.  INTRODUCTION 

The  introduction  and  extension  of  the  use  of  Information  and  Communications  Technology  (ICT)  has  been  of 
concern  in  the  educational  field  over  recent  years,  as  can  be  evidenced  by  studies  of  change  management  and 
ICT  integration  in  schools  internationally  (e.g.  Wong  & Li  2006;  Owston  2007).  It  is  evident  that  the  aim  of 
implementing  educational  change  of  introducing  and  extending  the  use  of  ICT  in  classes  is  to  improve  the 
existing  teaching  practices.  However,  the  processes  of  managing  pedagogical  innovations  involving  ICT  in  a 
school  context  are  usually  fraught  with  difficulties  in  practice  in  many  other  countries  (Owston  2007).  That  is, 
whilst  the  significance  of  ICT  in  education  is  acknowledged,  practical  challenges  of  successfully  integrating 
ICT  into  the  curriculum  in  school  settings  also  arise.  More  specifically,  the  studies  concerning  successful  ICT 
implementation  in  schools  identify  that  the  key  and  shared  attribute  of  these  schools  lies  in  the  joint  lead  and 
active  involvement  of  the  headteacher  and  other  staff  members  in  leadership  positions  in  the  change  process 
of  ICT  development  (Tearle  2003;  Sheppard  2003)  Notably,  however,  even  though  new  teaching  approaches 
are  adopted  in  classroom  practices  at  the  initial  stage  of  the  change  process,  very  few  are  able  to  become 
sustainable  and  institutionalised  (Hargreaves  & Fink  2006).  Furthermore,  it  is  generally  accepted  that  making 
pedagogical  innovations  concerning  ICT  is  more  complicated  than  managing  changes  in  many  other  subject 
areas  (Fox  2003). 

Given  the  above  context,  the  main  purpose  of  the  present  study  is  to  examine  the  key  determinants  of  a 
school’s  success  in  managing  and  sustaining  pedagogical  innovations  in  ICT  implementation,  with  a specific 
focus  on  the  effect  of  leadership  approaches  in  the  change  process.  Centring  on  a specific  vocational  high 
school  in  Taiwan,  this  study  examined  the  way  in  which  the  entire  staff  succeeded  in  transforming  a 
traditional  school  into  an  acknowledged  ICT-capable  school.  Thus,  the  findings  can  show  the  patterns  of  the 
leadership  processes  of  managing  pedagogical  innovations  in  ICT  integration  in  school  settings  in  Taiwan. 
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2.  LITERATURE  REVIEW 

2.1  Sustainability  of  Educational  Change 

Sustainability  of  educational  change  typically  refers  not  only  to  the  durability  of  reform  movements  in  school 
settings,  but  also  to  the  fact  that  whether  these  movements  can  be  scaled  up  or  spread  from  a few  schools  to 
the  entire  educational  system  (Combs  2007;  Fullan  2006;  Hargreaves  & Fink  2006).  Therefore,  these  authors 
perceive  sustaining  changes  and  innovations  in  education  as  a much  tougher  and  more  complicated 
undertaking  than  simply  maintaining  educational  shifts  over  time.  For  Hargreaves  and  Fink  (2003), 
sustainable  educational  change  in  schools  has  five  critical  and  interrelated  features  below:  (1)  its  processes 
and  effects  are  deep,  broad  and  durable;  (2)  it  supports  continuous  learning  and  knowledge  sharing  for 
benefiting  everyone  in  the  educational  context,  and  thus,  it  does  not  profit  simply  a few  individuals  by  means 
of  partial  changes  and  improvements;  (3)  it  is  upheld  by  accessible  human  and  material  resources,  and  it 
develops  and  renews  its  resource  base;  (4)  it  does  not  result  in  any  negative  impact  on  its  surrounding 
environments;  and  (5)  it  promotes  diversity  and  emphasises  the  importance  of  learning  from  differences 
through  collective  efforts  for  constructive  debate  and  reflective  evaluation. 

From  the  perspective  of  systemic  transformation,  Fullan  conceptualises  sustainability  of  educational 
change  as  ‘the  capacity  of  a system  to  engage  in  the  complexities  of  continuous  improvement  consistent  with 
deep  values  of  human  purpose’  (Fullan  2005,  p.  ix).  Based  on  the  above  conceptualisation,  capacity-building 
for  educational  change  should  focus  not  merely  on  the  schooFcommunity  level,  but  on  the  district  level  and 
the  state/central  government  level  as  well  (Fullan  2005).  Apart  from  the  above  advocacy  of  large-scale  and 
deep  reforms  at  all  levels,  the  remaining  interconnected  elements  which  Fullan  considers  fundamental  for 
sustainable  educational  change  are  as  follows:  (1)  starting  educational  change  with  moral  purpose;  (2) 
enhancing  whole-school  capacity  for  educational  change  by  developing  learning  networks  and  communities 
across  different  schools;  (3)  developing  systemic  accountability  both  by  building  vertical  relationships 
between  the  central  government  and  school  districts,  and  by  enhancing  cross-level  communicative  avenues; 
(4)  forming  a prevailing  culture  of  deep  learning  for  coping  with  endless  challenges  in  the  processes  of 
educational  change;  (5)  pursuing  both  short-term  and  long-term  results  for  achieving  system-wide  and  lasting 
changes;  (6)  a cyclical  mode  of  two  counterbalancing  forces  - activity  and  rest  - is  the  component  for 
stepping  into  successful  systemic  change  in  education;  and  (7)  sustaining  educational  change  requires 
continuation  of  adequate  leadership  capacity  which  is  built  throughout  the  organisation  at  all  levels  of  the 
system.  Notably  however,  while  stressing  the  importance  of  deep  and  large-scale  reform  of  the  holistic 
educational  system,  Fullan  (2005)  views  successful  leadership  at  the  school  level  as  the  primary  and  essential 
bedrock  of  integrating  the  above  requirements  for  sustaining  educational  change  into  school  contexts.  Indeed, 
although  noting  the  potential  impact  of  a school’s  surrounding  environments  on  the  sustainability  of 
educational  change,  Hargreaves  and  Fink  (2006)  still  highlight  the  fact  that  leadership  capacity  within  a 
school  is  the  radical  determinant  of  continuity  or  discontinuity  of  educational  change. 

It  is  evident  that  Fullan,  Hargreaves  and  Fink  argue  sustainability  of  educational  change  from  different 
perspectives;  however,  they  reach  a consensus  that  the  ways  in  which  schools  react  to  reform  agendas 
generally  decide  whether  or  not  educational  innovations  are  able  to  continue  and  become  system-wide.  Their 
common  opinion  raises  the  question  of  the  way  in  which  schools  develop  their  capacities  for  acting  positively 
and  effectively  on  continuous  reform  movements.  Like  Fullan,  Hargreaves  and  Fink,  other  researchers’ 
shared  response  to  the  above  question  is  that  schools  must  become  learning  organisations  which  cultivate 
staff  collaboration  and  continuous  professional  development  (Combs  2007;  Leithwood  2005).  These 
researchers’  arguments  also  echo  the  advocacies  by  Fullan,  Hargreaves  and  Fink  that  effective  school 
leadership  is  the  core  of  building  school  capacity  for  coping  with  changes,  and  that  developing  sustainable 
school  leadership  is  the  fundamental  and  key  strategy  for  sustaining  educational  change  and  improvement. 
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2.2  School  Leadership  for  Sustaining  Good  Practices 

‘Setting  directions,  redesigning  the  organisation  and  developing  people’  are  identified  by  Leith  wood  and 
Riehl  (2003)  as  the  crucial  principles  of  successful  school  leadership.  In  his  later  international  studies, 
Leithwood  (2005)  goes  further,  verifying  ‘developing  people’  as  the  key  determinant  of  making  good 
practices  durable  and  wide-spreading  in  the  educational  context.  For  Leithwood  and  Riehl,  the  principle  of 
developing  people  concerns  the  following  areas:  (1)  providing  intellectual  stimulation;  (2)  emphasising 
individualised  needs  and  professional  learning;  and  (3)  setting  appropriate  models  which  are  consistent  with 
the  school’s  values  and  goals.  The  successful  headteacher  supplies  professional  training  to  enhance  teachers’ 
skills,  and  created  a school  culture  which  nourishes  the  constructive  debate  and  open  evaluation  among  staff. 

The  initiation  of  school  change  usually  requires  teachers’  sufficient  competency  for  adapting  themselves 
to  reform  and  their  aspiration  for  educational  improvement  (Fullan  2001).  Importantly,  the  encouragement 
and  support  from  the  headteacher  could  be  the  main  force  for  driving  teachers  to  participate  in  continuous 
professional  learning.  In  their  case  studies  across  different  schools  in  England,  Leithwood  and  his  colleagues 
found  that  all  headteachers  had  a strong  and  positive  impact  on  the  staffs  motivation,  commitment  and 
beliefs  about  the  supportiveness  of  their  working  conditions  (Leithwood  et  al.  2006).  They  concluded  that  the 
headteacher  is  the  crucial  supporter  for  staff  to  develop  their  personal  mastery  in  embarking  on  school  change. 
More  specifically,  successful  headteachers  are  able  to  lead  staff  in  considering  the  values  and  reflecting  upon 
the  action  for  new  educational  agendas  in  a critical  and  constructive  way  (Leithwood  et  al.  2006). 

The  studies  of  sustaining  educational  change  in  schools  address  that  the  value  of  teachers’  collaborative 
learning  lies  in  showing  their  respect  for  and  learning  from  individuals’  diversity  (Fullan  2006;  Hargreaves  & 
Fink  2006).  It  is  for  these  reasons  that  the  headteacher’s  and  other  senior  leaders’  prompt  mediation  of 
turning  conflicts  among  staff  members  into  productive  discourse  and  discussions  is  particularly  imperative  in 
the  organisational  processes.  That  is,  without  the  competent  headteacher  and  other  senior  leaders  to  manage 
the  conflicting  tensions  among  staff  members,  individuals’  diverse  opinions  and  critical  dialogue  cannot  be 
beneficial,  but  fragmented  or  even  harmful  for  people  who  work  together  in  pursuit  of  school  change  and 
improvement  (Fullan  2006).  Moreover,  educational  policies  are  usually  overload  and  fragmented,  and  this 
can  result  in  disturbance  as  schools  are  asked  to  carry  out  educational  change  (Fullan  2001;  Hargreaves  2002). 
Even  so,  teachers’  resistance  to  implementing  and  sustaining  edueational  change  can  be  diminished,  when  the 
headteacher  enlarges  the  sehool’s  capaeity  by  supporting  teaehers  in  both  rational  and  emotional  ways 
(Eullan  2001).  It  would  appear  that  the  problems  with  educational  policies  can  cause  teachers  to  struggle  with 
managing  school  change.  However,  compared  with  external  factors,  the  internal  factor  - the  way  in  which  the 
headteacher  shapes  the  common  values  among  teachers  - tends  to  be  more  influential  to  educational  reform 
within  the  school  context. 

It  was  also  worth  noting  that  continuation  of  developing  leadership  talent  on  the  staff  and  collaborative 
leadership  are  usually  treated  as  the  critical  components  of  successfully  initiating  and  persisting  in  good 
practices  (Eullan  2006;  Hargreaves  & Eink  2006).  As  with  these  authors’  work,  the  studies  of  school 
leadership  for  change  management  consistently  reinforce  the  powerful  impact  of  leadership  distribution  to 
teachers  on  continuous  improvements  in  teaching  practices  (Spillane  2006).  More  recent  international 
research  of  ICT  integration  in  school  settings  also  verified  staff  collaboration  in  the  leadership  processes  as 
the  key  to  a school’s  sustainability  of  implementing  ICT  (Owston  2007).  Corresponding  to  Owston’s 
research,  the  studies  concerning  school  change  of  ICT  implementation  in  Canada  (Sheppard  2003),  England 
(Tearle  2003)  and  Hong  Kong  (Wong  & Li  2006)  revealed  that  the  tasks  of  pedagogical  innovations  in  ICT 
integration  should  not  be  put  on  the  shoulders  of  one  particular  school  leader,  but  be  appropriately  distributed 
to  the  staff  at  different  levels  if  educational  change  of  ICT  integration  is  to  be  successful  and  sustainable. 

All  the  above  studies  could  be  summarised  as  saying  that  if  school-wide  pedagogical  innovations  are  to 
be  successful  and  sustainable,  school  leaders  are  required  to  take  the  responsibilities  for  nurturing  the 
potential  leaders  on  school  staff  and  for  making  a wholesale  shift  in  teachers’  attitudes  towards  the  reform 
movement.  In  other  words,  not  only  the  formal  leaders’  high  commitment  to  ICT  development,  but  other  staff 
members’  joint  contribution  to  the  leadership  processes  of  implementing  ICT  is  also  fundamental  for 
managing  wholes-school  and  endurable  pedagogical  innovations  in  ICT  adoption. 
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3.  METHODOLOGY 

In  the  methodological  literature,  many  authors  have  made  similar  comments  that  mixing  quantitative  and 
qualitative  approaches  in  the  same  research  design  is  feasible  (Robson  2002).  Moreover,  incorporating 
quantitative  and  qualitative  research  approaches  and  methods  into  a single  study  is  particularly  instrumental 
for  further  exploration  of  social  phenomena  (Denscombe  2003).  Owing  to  the  usefulness  and  feasibility  of 
combining  both  quantitative  and  qualitative  approaches  in  one  study,  a mixed-method  approach  was  used 
within  the  present  study  for  investigating  the  target  school’s  leadership  approaches  to  pedagogical 
innovations  in  ICT  integration.  Purposeful  sampling  was  applied  to  ensure  that  the  school  selected  for  this 
research  was  an  information-rich  site.  The  target  school  was  a publicly  acknowledged  ICT-capable 
vocational  high  school  and  evaluated  by  the  government  as  being  qualified  for  continuing  running  the  change 
project  concerning  pedagogical  innovations  of  ICT  integration.  Findings  presented  in  this  paper  were 
collected  through  questionnaires  and  semi-structured  interviews.  The  questionnaires  were  distributed  to  30 
teaching  staff  and  a total  of  25  returns  were  achieved;  the  return  rate  was  83%.  All  the  respondents  to  the 
questionnaires  were  asked  to  give  their  answers  by  registering  on  a six-point  bipolar  scale  ranging  from  ‘very 
strongly  agree’  to  ‘very  strongly  disagree’.  The  levels  of  ‘very  strongly  agree’,  ‘strongly  agree’,  ‘agree’, 
‘disagree’,  ‘strongly  disagree’  and  ‘very  strongly  disagree’  were  translated,  respectively,  into  the  scores  6,  5, 

4,  3,  2 and  1 in  the  process  of  data  analysis.  As  regards  the  follow-up  interviews,  data  was  gathered  from  a 
total  of  19  school  staff,  including  the  headteacher,  the  director  of  academic  affairs,  the  ICT  coordinator  and 
16  teachers. 


4.  FINDINGS 

4.1  Findings  from  the  Questionnaires 

Statements  presented  in  the  questionnaires  were  designed  for  examining  the  staff  opinions  on  their  leadership 
approaches  to  managing  school-wide  pedagogical  innovations  in  ICT  adoption  in  the  existing  teaching 
practices.  Totalling  all  questionnaire  responses  to  the  statements,  nearly  all  (97%)  fell  within  three  levels 
from  ‘agree’  to  ‘very  strongly  agree’,  with  an  overall  mean  of  4.52  (at  the  level  of  ‘strongly  agree’). 

In  analysing  all  answers  in  detail,  two  statements  were  rated  particularly  highly.  One  of  the  statements 
was  ‘there  is  coordinated  action  across  the  staff  at  all  levels  in  the  leadership  processes  of  pedagogical 
innovations  in  ICT  integration’.  The  other  statement  was  ‘there  is  a good  approach  to  developing  teachers’ 
leadership  potential  for  managing  school  changes  and  improvements  in  ICT  integration’.  According  to  the 
findings,  general  overviews  of  all  responses  to  the  two  statements  were  classed  as  the  level  of  ‘strongly 
agree’.  The  former  statement  attracted  100%  positive  responses,  over  half  (52%)  of  which  fell  within  two 
levels  from  ‘very  strongly  agree’  to  ‘strongly  agree’.  In  addition,  this  statement  received  the  top  individual 
mean  of  4.56.  On  this  basis,  there  seemed  to  exist  collegiate  work  patterns  in  leadership  practices  of 
undertaking  pedagogical  innovations  in  ICT  integration.  As  regards  the  latter  statement,  nearly  all  responses 
(96%)  were  positive  and  around  half  (48%)  fell  within  two  levels  from  ‘very  strongly  agree’  to  ‘strongly 
agree’.  This  statement  was  ranked  the  second  highest  in  the  list  of  individual  means.  This  result  reflected  that 
most  staff  agreed  with  their  in-house  mechanisms  for  cultivating  the  talented  individuals  as  their  future 
leaders  in  the  domain  of  ICT  developments.  Compared  with  the  above  statements,  the  statement  concerning 
teachers’  satisfaction  with  the  overall  approach  to  school  leadership  for  pedagogical  innovations  in  ICT 
integration  was  ranked  slightly  lower.  Even  so,  the  general  overview  of  the  responses  to  this  statement  was 
still  relatively  positive,  with  a 96%  approval  rating.  Therefore,  it  can  be  said  that  nearly  all  staff  subscribed  to 
the  shared  views  that  their  overall  approach  to  school  leadership  was  satisfactory,  in  terms  of  undertaking 
pedagogical  innovations  in  ICT  integration. 

Based  on  these  figures,  there  was  a strong  tendency  reflecting  that  the  teachers’  opinions  on  their 
leadership  approaches  to  implementing  ICT  across  the  curriculum  were  positive.  Furthermore,  it  can  be 
assumed  that  school  leadership  for  pedagogical  innovations  in  ICT  adoption  was  not  restricted  to  a limited 
staff  in  leadership  positions  or  with  specifically  strong  skills  in  ICT  at  the  implementation  stage  of  ICT 
development. 
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4.2  Findings  from  the  Interviews 

4.2.1  Collaborative  Leadership  Approach 

Corresponding  to  the  questionnaire  results,  a particularly  strong  theme  which  emerged  in  the  interview  phase 
was  that  the  teachers  generally  felt  pleased  with  working  together  and  getting  involved  in  leadership  practices 
of  implementing  pedagogical  innovations  in  ICT  integration.  All  teachers  who  were  interviewed  expressed 
the  same  views,  arguing  that  their  school  leadership  was  not  treated  as  the  prerogative  of  any  individual  staff. 
In  the  interviews  with  the  teachers  from  outside  the  leadership  team,  their  common  opinions  were  that  they 
were  becoming  increasingly  comfortable  with  taking  the  leadership  role  in  the  processes  of  school-wide 
pedagogical  innovations  in  ICT  integration.  An  inexperienced  teacher,  for  example,  appreciated  that: 

This  school  is  like  a family  and  the  morale  is  very  high...  Although  I was  not  in  the  leadership 
team,  many  teachers,  including  formal  leaders  like  the  ICT  coordinator  and  directors,  invited 
me  to  join  their  team  discussion  and  decision-making  processes. ..It  was  really  heart- 
warming that  everyone  here  tried  making  me  feel  accepted. 

In  the  interviews  with  the  headteacher,  he  claimed  that: 

Without  the  continuous  and  joint  efforts  of  the  director  of  academic  affairs,  the  ICT 
coordinator  and  other  teachers  in  carrying  out  school-wide  pedagogical  innovations,  it 
would  have  been  almost  impossible  to  allow  our  school  to  have  today ’s  outcomes. 

Speaking  of  the  key  drivers  for  the  staff  collaboration  and  active  engagement  in  school  leadership  for 
pedagogical  innovations  in  ICT  integration,  all  teachers  in  the  interview  phase  claimed  that  the  headteacher 
was  in  the  central  role  in  shaping  a school  culture  within  which  individual  staff  became  used  to  coordinating 
with  each  other  in  undertaking  the  leadership  tasks  of  promoting  ICT  integration.  More  specifically,  the 
responses  from  all  teachers  reached  the  similar  comments,  highlighting  the  fact  that  shouldering  the 
leadership  tasks  offered  them  more  opportunities  of  working  closely  with  senior  and  middle  leaders  in 
proposing  and  revising  the  strategies  for  making  new  practices  fit  well  with  the  existing  classroom  practices. 
The  teachers,  therefore,  felt  that  getting  involved  in  the  leadership  activities  strengthened  their  confidence  in 
conducting  new  teaching  practices  of  ICT  adoption. 

4.2.2  Development  of  Future  Leaders 

In  the  interview  phase,  the  staff  expressed  their  positive  views  on  the  question  about  their  in-house 
mechanisms  for  developing  teachers’  leadership  potential  to  implement  ICT.  For  all  teachers  in  the 
interviews,  the  school’s  initial  and  current  achievements  in  ICT  implementation  lay  in  the  headteacher’s  good 
appointment  of  the  suitable  teachers  as  the  key  leaders  for  whole-school  development  in  the  field  of  ICT 
education  and  its  application.  All  teachers  made  the  similar  comments,  stating  that  their  ICT  coordinator  was 
very  ICT-focused,  highly-committed  and  had  sufficient  interpersonal  skills.  For  example: 

Our  ICT  coordinator  is  very  enthusiastic  about  helping  us  out  whenever  we  have  trouble  with 
the  computer.  He  is  like  our  total  guide  with  ICT. 

Our  headteacher  is  quite  visionary  and  has  well-targeted  development  plans  for  managing 
human  resources...  He  selected  ‘the  right  person’  as  the  ICT  coordinator.  I think  this  is  the 
important  base  for  our  school ’s  outcomes  of  ICT  implementation. 

In  addition  to  good  designation  of  the  key  staff  in  the  very  beginning  of  the  change  process,  the 
headteacher’s  investment  in  fostering  the  future  leaders  for  sustaining  ICT  implementation  was  the  crucial 
issue  which  was  raised  frequently  and  positively  in  the  interview  phase.  Apart  from  this,  it  was  worth  noting 
that  the  strong  support  from  another  formal  leader  - the  director  of  academic  affairs  - in  the  overall  course  of 
developing  individuals’  potential  to  manage  school  changes  in  ICT  adoption.  This  is  because  when  praising 
the  headteacher’s  efforts  to  cultivate  leadership  abilities  of  individual  staff,  the  teachers  also  highlighted  the 
director  of  academic  affairs’  hands-on  approaches  to  leading  others  by  examples.  As  can  be  seen  in  the 
teachers’  replies: 
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The  director  of  academic  affairs  eased  our  nerves  by  showing  us  the  practical  strategies  for 
leading  others  and  managing  the  project...!  think  that  he  is  really  competent  for  “leading  us 
to  understand  how  to  lead  ’’ 

I think  both  the  headteacher  and  the  director  of  academic  affairs  are  very  supportive  for 
what  we  do... The  headteacher  allows  us  to  exert  leadership  in  the  field  which  we  are  familiar 
with... The  director  he  coaches  us  by  showing  how  to  do  and  how  to  make  things  better.  I 
think  he  is  an  excellent,  a very  approachable  leader. 

In  the  interview  with  the  director  of  academic  director,  he  said  that: 

I’ve  never  seen  making  whole-school  pedagogical  innovations  as  an  easy  job. ..Of  course, 
making  effective  change  is  even  more  challenging. ..However,  being  a leader,  you  need  to  do 
your  best  to  comfort  teachers’  nerves. ..It  is  my  belief  that  you  need  to  show  them  the  way 
how  to  embark  on  new  projects  and  share  some  experiences  with  them  in  advance. 

Based  on  the  above  interviewees’  responses,  it  was  evident  that  in  order  to  construct  the  supportive 
conditions  of  implementing  and  sustaining  pedagogical  innovations  in  ICT  integration,  the  headteacher 
invested  much  energy  in  identifying  the  competent  individuals  as  the  core  leaders  at  the  very  start  of  the 
change  process.  Apart  from  this,  the  headteacher  and  the  director  of  academic  affairs  both  provided  adequate 
support  for  the  ICT  coordinator  in  the  aspect  of  nurturing  and  renewing  human  resources  within  the  school 
throughout  the  change  process.  Moreover,  it  was  encouraging  to  note  that  for  those  from  outside  the 
leadership  team,  getting  involved  in  leadership  practices  was  perceived  to  be  beneficial  for  future  career 
plans.  The  positive  feelings  about  exercising  leadership  in  the  change  process  seemed  to  motivate  the  staff  to 
become  more  willing  to  participate  in  leadership  activity  involving  ICT  implementation. 

4.2.3  Perceived  Values  in  Pedagogical  Innovations 

In  analysing  the  interview  data,  94%  of  the  teachers  approved  their  school  leadership  for  managing  changes 
of  ICT  integration  without  dissent.  When  further  explaining  why  they  felt  satisfied  with  the  overall  approach 
to  school  leadership  for  pedagogical  innovations  in  ICT  integration,  the  interviewees’  responses  can  be 
divided  into  two  key  points.  First,  the  success  in  gaining  official  recognition  of  initiating  and  continuing  the 
change  project  allowed  the  staff  to  receive  more  support  from  the  government  to  develop  and  improve  the 
ICT-integrated  instructional  modes.  This,  in  turn,  yielded  the  benefits  for  the  teaching  and  learning  processes. 
Second,  the  school’s  image  and  competitive  capabilities  improved  throughout  the  course  of  whole-school 
developments  in  new  practices  of  ICT  application.  In  the  interviews,  the  teachers  claimed  that: 

We  felt  exhausted  at  the  very  beginning  of  undertaking  the  change  project  of  ICT  integration, 
but  now  we  ’re  at  a point  that  we  are  proud  of  being  part  of  the  staff  here...  Our  neighbouring 
schools  know  that  integrating  ICT  into  the  curriculum  has  become  the  key  feature  of  our 
school.  We  enjoy  a good  image  within  our  school  community. 

The  overwhelming  majority  of  the  teachers  (94%)  also  considered  their  leadership  approaches  to  be 
successful  in  nearly  every  aspect  of  handling  pedagogical  innovations  and  developments,  not  simply  in  the 
aspect  of  implementing  ICT  across  the  curriculum.  Moreover,  the  interview  results  showed  that,  when  noting 
the  things  which  the  school  could  change  and  improve,  the  teaching  staff  were  keen  on  giving  their  voice  in 
the  staff  meeting  or  even  proposed  their  ideas  directly  to  the  headteacher  or  other  formal  leaders. 


5.  DISCUSSION 

With  respect  to  the  leadership  approach  to  managing  changes  of  ICT  implementation,  it  was  evident  that 
leadership  tasks  of  maximising  the  use  of  ICT  for  teaching  purposes  were  fulfilled  through  considerable 
synergies  and  interactions  of  many  staff  members,  irrespective  of  post  or  ICT  expertise.  That  is,  there  seemed 
to  exist  a distributed  form  of  leadership  approaches  to  implementing  ICT  in  the  target  school.  The  evidence 
reinforced  the  conclusions  of  the  related  studies  (Wong  & Li  2006)  that  teachers’  coordinated  actions  and 
active  engagement  in  the  leadership  processes  were  at  the  heart  of  making  school-wide  pedagogical 
innovations  in  ICT  integration  successful  and  durable.  The  findings  also  resonate  with  Tearle’s  case  studies 
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(2003)  that  teachers’  collaboration  in  shouldering  leadership  responsibility  in  the  change  process  was  at  the 
core  of  enlarging  their  school’s  leadership  capacity  for  continuing  pedagogical  innovations  in  ICT 
implementation.  The  findings  also  support  Leithwood’s  (2005)  international  multi-case  studies  that  teachers’ 
intense  involvement  and  close  collaboration  in  leadership  activities  were  found  to  be  essential  for  effective 
leadership  which  brought  about  successful  changes  and  improvements  in  school  settings  in  nearly  all 
educational  contexts.  Importantly,  teachers  in  the  target  schools  generally  accepted  their  collaboration  and 
collegiality  in  the  leadership  processes  of  implementing  ICT  as  a natural  and  an  important  part  of  their  work 
routine.  To  a certain  degree,  the  findings  echoed  Sheppard’s  multi-case  studies  (2003)  that  the  predominant 
determinant  of  the  schools’  success  in  continuing  new  teaching  practices  of  ICT  integration  was  teachers’ 
awareness  of  the  necessity  and  value  of  their  collaboration  in  leadership  practices. 

In  the  aspect  of  developing  school  staff  for  sustaining  ICT  implementation,  the  findings  showed  that  the 
headteacher’s  comprehension  of  the  staffs  quality,  together  with  good  designation  of  the  key  leaders,  seemed 
to  be  essential  for  success  in  planning  and  initiating  whole-school  ICT  developments.  The  evidence  gathered 
here  was  parallel  to  the  related  studies  focusing  on  implementing  school-wide  changes  in  pedagogical 
innovations  regarding  ICT  adoption  in  Canada  (Sheppard  2003),  England  (Tearle  2003)  and  Hong  Kong 
(Wong  & Li  2006).  Despite  differences  in  their  educational  contexts,  these  studies  subscribed  to  the  same 
views,  arguing  that  identifying  and  developing  the  competent  teacher  as  the  ICT  coordinator  (or  the 
‘technology  teacher’  in  Sheppard’s  terms)  was  the  foremost  requirement  for  bringing  about  the  initial  success 
in  the  change  process  of  school-wide  ICT  adoption.  Notably,  both  the  headteacher  and  another  senior  leader 
(e.g.  the  director  of  academic  affairs)  continued  nurturing  the  potential  staff  as  lead  teachers  in  the  ICT  field. 
Moreover,  echoing  the  literature  on  sustainable  leadership  and  school  change,  the  results  from  this  study 
reflected  that  the  headteacher’s  competence  in  identifying  and  continuing  fostering  individuals’  leadership 
potential  may  function  as  the  primary  base  for  making  the  initial  success  in  school  change  become  durable 
(Fullan  2006;  Hargreaves  & Fink  2006).  In  a sense,  it  would  appear  that  only  with  the  long-term  plans  and 
systemic  strategies  for  enhancing  leadership  capacities  of  many  staff  members  can  a school  have  sufficient 
capacity  for  moving  the  change  and  improvement  efforts  beyond  the  initial  success  to  the  continuation  phase. 

As  for  teachers’  perception  of  pedagogical  innovations,  it  was  encouraging  to  note  that  the  interviewees 
generally  approved  their  school  leadership  for  managing  changes  of  ICT  integration  without  dissent.  Apart 
from  this,  the  teachers  considered  their  leadership  approaches  to  be  successful  in  nearly  every  aspect  of 
handling  pedagogical  innovations  and  developments,  not  simply  in  the  aspect  of  implementing  ICT  across 
the  curriculum.  More  specifically,  when  noting  things  which  the  school  could  change  and  improve  for  better, 
the  teachers  were  keen  on  giving  their  voice  in  the  staff  meeting  or  even  proposed  their  ideas  directly  to  their 
formal  leaders.  On  this  basis,  an  overarching  message  which  emerged  in  the  school  was  that  the  rationale  for 
the  teachers’  high  praise  for  their  school  leadership  lay  in  their  strong  awareness  of  the  ongoing  and  whole - 
school  progression  throughout  the  change  process.  Based  on  the  findings,  it  could  be  inferred  that  one  of  the 
radical  impulses  for  school  staff  to  continue  moving  the  existing  success  in  ICT  implementation  forward  was 
that  the  school  fostered  a dominant  culture  within  which  people  held  a high-level  commitment  to  striving  for 
excellence.  That  is,  teachers’  awareness  of  the  necessity  for  school  change  underpinned  their  intentions  to 
keep  improving  in  ICT  implementation  even  when  challenges  occurred.  The  findings  correspond  to  the 
literature  that  school  staff  are  usually  willing  to  undertake  change  and  development  when  feeling  a critical 
need  for  doing  so  (Fullan  2001). 


6.  CONCLUSION 

Despite  a small  number  of  research  participants  in  the  present  study,  the  combination  of  quantitative  and 
qualitative  approaches  provides  insights  into  school  leadership  for  managing  pedagogical  innovations  in  ICT 
implementation.  The  factors  which  affected  the  school’s  sustainability  of  pedagogical  innovations  in  ICT 
implementation  are  inter -related  rather  than  discrete.  Yet,  it  would  appear  that  collaborative  leadership  is  at 
the  core  of  supporting  whole-school  change  and  improvement  in  this  regard.  In  addition,  the  results  show  that 
a school’s  capacity  for  sustaining  ICT  implementation  entails  not  only  good  designation  of  the  suitable  staff 
as  the  key  leaders  at  a very  start,  but  also  ongoing  development  of  the  potential  teachers’  leadership  abilities. 
The  findings  also  identify  the  impact  of  school  leadership  on  teachers’  perceived  values  in  pedagogical 
innovations  which,  in  turn,  affects  their  participation  in  change  management  concerning  ICT  implementation. 
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It  is  inevitable  that  the  findings  from  one  case-study  school  limit  the  possibilities  for  generalisation.  However, 
little  research  is  undertaken  to  examine  the  change  process  focusing  exclusively  on  school  leadership  for 
sustaining  good  practices  of  ICT  integration.  It  is  anticipated  that  the  findings  from  this  study  can  shed  light 
on  the  approaches  to  managing  and  sustaining  pedagogical  innovations  in  ICT  integration.  Thus,  it  is 
expected  that  the  result  of  the  present  research,  on  the  one  hand,  contributes  to  illuminating  the  potential 
factors  which  facilitate  sustainability  of  school  change  and  improvements  involving  ICT  adoption.  On  the 
other  hand,  it  can  extend  knowledge  in  the  field  of  school  leadership  for  change  management  regarding  ICT 
implementation. 
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ABSTRACT 

Now  days  the  growing  need  for  highly  qualified  computer  science  educators  in  modem  educational  environments  is 
commonplace.  This  study  examines  the  potential  use  of  Greek  School  Network  (GSN)  to  provide  a robust  and 
comprehensive  e-training  course  for  computer  science  educators  in  order  to  efficiently  exploit  advanced  IT  services  and 
establish  a modem  and  versatile  education  environment  in  the  Greek  Society.  Furthermore,  a preliminary  questionnaire 
survey  was  performed  in  order  to  validate  the  adoption  of  GSN  and  also  to  formulate  a realistic  training  course 
customized  to  specific  and  future  needs  of  computer  science  educators  in  primary  and  secondary  education,  thus  enabling 
them  to  immerse  in  real-world  situations  (i.e.,  school  computer  laboratory).  Findings  from  this  preliminary  survey  are 
also  presented. 
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tele-education,  Greek  School  Network  (GSN),  training,  computer  science  educators 


1.  INTRODUCTION 

The  Ministry  of  Education*  (MoE)  has  established  the  strategic  plan  for  the  “Digital  School”,  in  order  to 
modernize  the  education  system  through  the  use  of  the  ICTs.  In  the  context  of  the  “Digital  School”,  on-going 
and  near  future  actions  in  both  primary  and  secondary  education  evolve  the  wide  use  of  innovative 
technologies  (i.e.,  Web  2.0  applications  tools),  in  order  to  enhance  the  traditional  and  lecture-dominant 
teaching  methods  and  provide  a flexible  and  attractive  learning  environment.  Related  actions  include; 

• Initiation  of  "Bring  You  Own  Device"  (BYOD)  methodology  to  primary  and  secondary  schools. 

• Installation  of  mobile  computer  laboratories  with  enhanced  functionalities  (i.e.,  broadband  wireless 
access,  multimedia  equipment)  and  integration  of  advanced  devices  such  as  interactive  whiteboards. 

• Schools  interconnection  to  municipal  optical  metropolitan  area  networks,  enabling  high-speed 
optical  access  to  GSN. 

• Interactive  and  experimental  learning  environment  considering  the  use  of  (a)  tablets  or  e-book 
readers,  (b)  low  cost  electronic  prototyping  platforms  (i.e.,  Arduino  kit,  Raspberry  Pi),  (c)  3D 
printers  and  (d)  programmable  robotic  and  artificial  intelligent  systems. 

• 3D  visualization  and  interaction  [1]. 

In  order  the  above  activities  to  achieve  their  potential  goals,  an  updated  and  targeted  training  course 
should  be  provided  to  computer  science  educators.  This  will  establish  a flexible  framework  that  will  further 
promote  the  utilization  of  information  and  communication  technology  infrastructure  in  schools.  The  initial 
design  and  the  conceptual  framework  of  the  training  course  was  based  on  well-established  practices  [2], 
while  further  adjustments  and  improvements  will  be  performed  taking  account  the  findings  of  an  electronic 
questionnaire  survey  appropriate  formulated  [3].  The  e-training  course  will  be  realized  using  the  Greek 
School  Network  (GSN)  [4].  This  was  an  optimal  solution  since  GSN  delivers  high  quality  and  a wide  range 
of  network  and  telematics  services  to  its  end  users  [4-6]. 
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This  paper  is  organized  as  follows:  In  Section  2 an  overview  of  GSN  core  and  advanced  services  is 
presented.  In  Section  3 the  basic  scheme  and  functionality  of  the  information  system  for  management  and 
monitoring  the  tele-education  progress  is  briefly  described.  In  Section  4 the  conceptual  framework  of  the 
training  course  is  analyzed,  while  in  Section  5 the  evaluation  of  the  electronic  questionnaire  survey  is 
presented.  Finally,  conclusions  are  drawn  in  Section  6. 


2.  GSN  CORE  AND  ADVANCED  SERVICES  OVERVIEW 

The  Greek  School  Network^  (GSN)  was  founded  in  1998  and  constitutes  the  educational  intranet  of  the 
Ministry  of  Education  [4].  GSN  interconnects  all  Greek  schools,  educational  administration  offices  and 
organizations  and  also  provides  advanced  e-services  to  students,  teachers,  and  school  administrator 
communities  [5].  It  was  funded  by  the  second  and  third  Community  Support  Framework  of  the  European 
Union  and  is  currently  maintained  and  supported  by  a group  of  twelve  research  centers  and  universities, 
under  the  management  of  Computer  Technology  Institute  and  Press  “Diophantus”  (CTIP)^  [6].  It  is  the 
biggest  national  public  network  according  to  the  number  of  users,  and  has  been  recognized  internationally  as 
a remarkable  educational  network  that  promotes  the  introduction  and  exploitation  of  Information  and 
Communication  Technologies  (ICT)  in  the  Greek  educational  system.  Furthermore,  due  to  its  innovative  role 
the  GSN  fulfills  the  scientific  and  research  goals  for  the  diffusion  of  Internet  access  and  the  promotion  and 
establishment  of  contemporary  educational  models. 

The  GSN  services  [4]  can  be  divided  into  the  following  categories:  (i)  Basic  services:  network 
connectivity,  user  authentication  and  authorization,  GSN  and  students  portal,  automatic  user’s  registration, 
(ii)  Communication  services:  email  (with  anti-virus  and  anti-spam  protection),  electronic  lists,  instant 
messaging,  video-teleconference,  (iii)  Web  hosting  and  blogging:  teacher  and  school  pages  hosting, 
authoring  tools  for  web  pages,  dynamic  pages  hosting,  database  service,  blogs,  (iv)  Advanced  services:  Video 
on  Demand  (VoD),  live  casting,  synchronous  and  asynchronous  e-learning,  e-portfolios,  social  networking, 
(v)  Complementary  services:  electronic  cards,  school  magazines  and  newspapers,  team  calendars,  file 
sharing,  (vi)  Central  infrastructure  services:  naming  service  (DNS),  directory  service  (LDAP),  distributed 
helpdesk,  online  statistics,  asset  management  and  (vii)  Management  services:  users  management  service, 
network  monitoring,  network  security,  remote  router  administration,  CERT,  statistics,  etc.. 

The  aforementioned  services  have  been  designed  to  appeal  to  all  GSN  users  offering  a versatile 
environment  suitable  for  the  implementation  of  modern  educational  methodologies  and  the  introduction  of 
modern  practices.  Traditional  teacher  centered,  lecture -dominant  and  text-based  teaching  methods  can  be 
enriched  and  become  more  attractive  using  GSN  services.  It  has  to  be  mentioned  that  the  role  of  GSN  is  not 
to  define  the  teaching  methods,  but  to  support  a coherent  set  of  well-established  educational  tools  that  can 
enhance  and  accommodate  modern  digital  education  methodologies.  This  implies  that  GSN  efficiently 
provides  a robust  information  and  communication  technology  framework  (i.e.,  high  quality  networking 
services)  allowing  large  scale  and  long-term  e-learning  and  training  activities,  as  is  described  in  the  following 
sections,  to  be  successfully  realized. 

As  described  below,  feedback  information  extracted  from  an  electronic  questionnaire  survey  indicates  a 
high  adoption  of  the  GSN  services  from  the  school  community  (i.e.,  teachers).  There  is  particularly  high 
utilization  of  electronic  services,  such  as  email,  emailing  lists,  websites,  blogs,  VoD  as  well  as  of  social 
networking  and  synchronous  and  asynchronous  e-learning  services.  Some  overall  statistics  (recorded 
February  2013)  validating  the  broad  use  of  GSN  services  are  depicted  in  Table  1. 


^ http://www.sch.gr 
^ http://www.cti.gr 
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Table  1 . Statistical  data  of  GSN 


GSN  services 

16 

Educational  websites 

12.747  (based  on  PHP,  Joomla,  Wordpress  or  ASP/.NET  technology) 

Educational  blogs 

19.223  (February  2013) 

Educational  communities 

318 

Digital  Courses  (current  year) 

4.786  courses  from  891  schools  of  Secondary  Education 

GSN  portal 

>340.000  unique  visitors  per  month 

Personal  accounts 

82.120  (teachers),  63.554  (students) 

Active  mailboxes 

>133.000 

Helpdesk  tickets 

>30. 000/year 

GSN  maintains  a thorough  tested,  reliable  set  of  services  that  exhibit  high  level  of  availability  and 
scalability  and  its  flexible  architecture  [4]  allows  the  integration  of  the  tele-education  platform  described 
below,  in  terms  of  management,  authentication  and  authorization. 


3.  TIME  SCHEDULING  AND  TELE-EDUCATION  PLATFORM 


A time  Scheduling  and  Tele-Education  Platform  (t-STEP)  was  designed  and  implemented  by  Greek  School 
Network  and  Networking  Technologies  Directorate'*  in  order  to  provide  a high  quality  and  flexible  training 
course  and  potentially  compensate  the  growing  needs  on  information  and  communication  technologies. 
Currently,  t-STEP  is  in  pilot  operation  and  is  evaluated  by  the  group  of  universities  and  research  institutes 
that  operate  and  maintain  the  GSN.  As  it  is  expected,  production  service  will  be  available  in  March  2013  at 
http;//training.sch.gr.  Considering  the  initial  design  framework,  t-STEP  comprises  the  following: 

• Frontend  module  (see  Figure  1):  Upon  successful  login  (i.e.,  valid  GSN  account)  time  scheduling 
(i.e.,  registration  in  forthcoming  e-lecture)  and  basic  administration  (i.e.,  update  users’  profile) 
operations  can  be  efficiently  performed.  Also,  some  statistical  data  can  be  extracted  (i.e.,  total  hours 
of  successfully  attended  e-lectures)  and  thoroughly  presented  to  the  user.  Furthermore,  a 
questionnaire  survey  is  provided  in  order  to  evaluate,  upon  completion,  the  overall  e-training 
process.  Additionally,  a link  to  a test-room  (virtual  class)  is  provided  to  verify  software  and 
hardware  compatibility,  in  order  to  successfully  participate  in  the  real-time  e-learning  environment 
described  below. 


Figure  2.  Front-end  web-page  of  the  training  platform 


• Admiuistratiou  (backeud)  module  (see  Figure  2):  The  backend  environment  provides  the  main 
administration  interface  of  the  t-STEP,  which  is  only  accessible  by  the  operators  of  the  GSN  and  the 


* http://www.cti.gr/nts 
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registered  instructors  (with  different  administrative  privileges).  Also,  a super  user  account  (i.e., 
supervisor)  is  provided  with  enhanced  administrative  functionalities  enabled.  Basic  operations  such 
as  monitoring  the  overall  progress  of  the  training  course  (i.e.,  successfully  completed  training 
sessions),  schedule  or  postpone  a training  task,  track  trainees  who  successfully  registered  and 
participated  in  a training  session,  extract  statistical  data  etc.  are  efficiently  performed  through  the 
backend  module. 


a 


2:  o □ 


Figure  2.  Administration  module  of  the  training  platform 

• Real-time  tele-education  environment  (see  Figure  3):  The  realization  of  each  training  session  is 
based  on  the  BigBlueButton^  open-source  software,  which  is  transparently  integrated  with  the  time- 
scheduling module  described  above.  In  principle,  the  administration  of  the  real-time  session  is 
performed  by  the  instructor  (i.e.,  moderator),  who  potentially  can  interact  with  the  participants  (i.e., 
mute/unmute  or  eject  user),  upload  slides  to  the  presentation  area  and  efficiently  coordinate  the 
related  learning  activities. 


Figure  3.  Typical  BigBlueButton  session. 

Following  the  initial  design  restrictions,  t-STEP  is  strongly  oriented  to  the  Greek  educational  system 
(i.e.,  primary  and  secondary  education).  Large  scale  deployment  of  the  training  course  described  below  will 
provide  critical  hints  for  potential  use  (considering  appropriate  adjustments)  in  related  training  and  life-long 
activities  and  also  for  advanced  educational  services  (i.e.,  university  post-graduate  courses). 


4.  CONCEPTUAL  FRAMEWORK  OF  THE  TRAINING  COURSE 

The  proposed  framework  of  the  training  course  defines  24  educational  topics  (ETs)  organized  under  four 
categories  shown  in  Table  2.  The  outline  of  the  proposed  training  framework  was  based  on  the  extensive  and 
fruitful  collaboration  between  the  administrators  of  the  GSN  [4-6]  and  the  hints  that  were  provided  by  the 
Users  Support  Service  (Help-Desk)®.  Furthermore,  useful  information  was  also  extracted  from  the  ticketing 
system’  utilized  by  the  Help-Desk. 

For  the  realization  of  the  intended  training  courses  and  the  in-depth  and  detailed  analysis  of  the  topics 
depicted  in  Table  2,  well-documented  resources  designed,  taking  into  account  the  adult  education  aspects  [7], 
including  (a)  lecture  notes,  complemented  by  practical  examples  and  case  studies,  (b)  instructor’s 


^ http://www.bigbluebutton.org/ 
* http://www.sch.gr/helpdesk 
’ http://helpdesk.sch.gr 
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presentation  slides,  (c)  research  topics  and  set  of  activities  meriting  further  investigation  and  practice,  (d) 
self-assignments  and  (e)  instructor’s  notes.  Typically  selecting  the  correct  mixture  of  available  resources  is  a 
challenging  task,  however  the  motivation  for  defining  the  logical  structure  of  the  resources  (i.e.,  reference 
material)  described  above,  was  to  fulfill  at  least  the  following  criteria  [2]: 

• Addressing  the  challenges  of  life  wide  and  lifelong  learning 

• Maximize  the  impact  of  distant  learning  providing  in-depth  training 

• Assessment  of  learning  and  the  provision  of  feedback 

• Enhance  learning  experience  and  learning  outcome 

• Expand  “educational  bandwidth”  limiting  time  constraints 

• Eacilitate  collaborative  work  and  online  discussion 

• Effective  content  and  course  delivery 

• Efficiently  address  both  technical  oriented  tasks  and  pedagogical  issues 

• Potentially  accommodate  growing  students  enrollment  in  digital  technologies 

• Address  the  growing  emphasis  on  technological  issues 

• Potentially  augment  central  support  services  considering  long-term  support  challenges 

• Strengthen  incentives  for  fostering  digital  technologies  and  encourage  innovation 


Table  2.  Conceptual  framework  of  the  proposed  training  course 


Categories 

Topics 

Organizational 

ETl  - The  role  of  computer  science  educator  in  the  context  of  “Digital  School” 
ET2  - Users  support  service  (helpdesk) 

Ministry  of  Education 
systems’  support 

ET3  - E-school  application 

School  Computer 
Laboratories 

ET4  - Administration  and  maintenance  of  school  computer  laboratories 

ET5  - Support  and  management  of  MS-Windows  based  operating  systems 

ET6  - Windows  2003  Server  administration 

ET7  - Windows  2008  Server  administration 

ET8  - ISA  server  2000  and  2004  administration 

ET9  - Squid  proxy/caching  server  administration 

ETIO  - Ubuntu  installation  and  configuration  in  school  laboratories 

ETll  - Open  source  educational  OS 

ET12  - Advanced  IT  devices  (i.e.,  interactive  whiteboards) 

Greek  School  Network 
Services 

ET13  - Email  service  utilizing  enhanced  security,  communication  & collaboration  features 

ET14  - Students  services 

ET15  - Open  source  educational  services 

ET16  - Web  hosting 

ET17  - Publishing  web  content  using  Joomla  CMS 

ET18  - Wordpress  configuration  and  management 

ET19  - Blogs  and  communities 

ET20  - E-class  and  e-learning  services 

ET21  - Moodle  Learning  Management  System 

ET22  - Internet  safety 

ET23  -Video  on  demand 

ET24  - Advanced  services  (i.e..  School  RSM) 

The  potential  interest  of  computer  science  educators  to  excel  their  technical  proficiency  in  the 
aforementioned  topics  was  examined  through  a questionnaire  survey  described  in  Section  5. 

5.  QUESTIONNARIE  SURVEY  EVALUATION 

The  final  version  of  the  questionnaire,  which  was  structured  after  fruitful  consultation  with  the  operators  of 
the  Greek  School  Network,  comprises  two  distinct  parts.  The  first  part  examines:  (i)  the  degree  of  awareness 
of  the  educational  community  for  the  electronic  services  provided  by  the  GSN,  (ii)  the  ICT  school 
infrastructure  and  (iii)  the  Users  Support  Services  (Help  Desk)  for  the  direct  support  regarding  the 
connectivity  to  GSN  or  its  electronic  services.  The  related  evaluation  tasks  are  listed  below: 
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• Q1  - Are  you  familiar  with  the  Greek  School  Network? 

• Q2  - How  well  informed  are  you  on  the  network  feasibility  (networking  access)  and  network 
bandwidth  provided  by  the  GSN  in  schools  and  administrative  units? 

• Q3  - Are  you  familiar  with  the  electronic  services  provided  by  the  Greek  School  Network? 

• Q4  - Are  you  taking  advantage  of  the  electronic  services  provided  by  the  Greek  School  Network? 

• Q5  - Do  you  know  where  you  can  access  documentation  and  official  guidelines  for  the  electronic 
services  provided  by  the  Greek  School  Network? 

• Q6  - How  well  informed  are  you  about  the  school  PC  laboratories  and  their  IT  infrastructure? 

• Q7  - Do  you  know  where  you  can  access  technical  documentation  for  school  PC  laboratories  and 
their  IT  infrastructure? 

• Q8  - How  well  informed  are  you  about  technical  support  services  for  troubleshooting  common  and 
advanced  IT  failures  at  school  PC  laboratories? 

The  second  part  of  the  questionnaire  records  the  potential  interest  of  the  computer  science  educators  to  be 
trained,  in  the  near  future,  in  a well-defined  set  of  technical  and  educational  topics,  as  was  described  in 
Section  4.  Its  organizational  structure  was  formally  based  on  the  conceptual  framework  shown  in  Table  2. 

For  the  realization  of  the  survey  a three-grade  scale  was  employed  [3],  depicted  in  Table  3,  while 
statistical  data  was  collected  by  means  of  an  electronic  questionnaire  administered  from  January  9th  until 
February  3rd  2013. 

Table  3.  Comparison  scale  for  evaluation  process 

Characterization 

None Sufficiently Considerably 

The  questionnaire  was  hosted  on  a GSN’s  web-server  and  posted  online  at  http://www.sch.gr/training. 
The  development  of  the  electronic  questionnaire  was  based  on  open  source  software  LimeSurvey^  v2.00-h, 
whereas  the  installation  and  configuration  was  performed  by  the  Greek  School  Network  and  Networking 
Technologies  Directorate.  In  order  to  successfully  login  to  the  above  link  a valid  GSN  account  was 
prerequisite,  while  the  estimated  time  for  the  questionnaire  completion  was  ten  minutes.  The  questionnaire 
was  pretested,  before  being  widely  distributed,  through  a pilot  study  which  helped  to  identify  and  address 
possible  ambiguities.  Thus,  suggestions  and  feedback  from  respondents  improved  the  final  structure  of  the 
survey.  A total  of  1469  educators  participated  to  the  questionnaire  survey  which  corresponds  to  19.13%  of 
the  overall  computer  science  educators  officially  registered  to  the  Ministry  of  Education.  Fifty-eight 
participants  gave  incomplete  answers,  thus  their  results  were  excluded  from  the  statistical  analysis.  Typical 
results  of  the  questionnaire  evaluation  considering  the  distinct  parts  described  above  are  shown  respectively 
in  Figures  4 and  5.  Furthermore,  a sorted  list  (in  descending  order)  of  the  e-training  topics  considering  the 
characterization  “Considerably”  (see  Table  3)  is  shown  in  Table  4. 


Figure  4.  Survey  quantitative  results  considering  GSN,  support  and  technical-oriented  tasks 


* http://www.limesurvey.org 


172 


lADIS  International  Conference  e-Learning  2013 


Figure  5.  Survey  quantitative  results  considering  technical  and  educational  topics  shown  in  Table  2 
Table  4.  A sorted  list  of  the  e-training  topics  considering  the  characterization  “Considerably” 


Topics  Considerably  (%) 

ET4  - Administration  and  maintenance  of  school  computer  laboratories  70.3 1 

ET16  - Web  hosting  68.10 

ET17  - Publishing  web  content  using  Joomla  CMS  67.55 

ET22  - Internet  safety  64.24 

ET15  - Open  source  educational  services  61.10 

ETll  - Open  source  educational  OS  60.05 

ETIO  - Ubuntu  installation  and  configuration  in  school  laboratories  58.78 

ET20  - E-class  and  e-leaming  services  58.58 

ETl  - The  role  of  computer  science  educator  in  the  context  of  “Digital  School”  56.87 

ET12  - Advanced  IT  devices  (i.e.,  interactive  whiteboards)  54.22 

ET5  - Support  and  management  of  MS-Windows  based  operating  systems  52.85 

ET21  - Moodle  Learning  Management  System  52.45 

ET24  - Advanced  services  (i.e..  School  RSM)  51.36 

ET19  - Blogs  and  communities  50.37 

ET9  - Squid  proxy/caching  server  administration  50.23 

ET14  - Student  services  47.89 

ET18  - Wordpress  configuration  and  management  47.89 

ET23  -Video  on  demand  46.59 

ET2  - Users  support  service  (helpdesk)  44.66 

ET7  - Windows  2008  Server  administration  41.45 

ET6  - Windows  2003  Server  adminisb'ation  40.1 1 

ET3  - E-school  application  38.96 

ET8  - ISA  server  2000  and  2004  administration  37.69 

ET13  - Email  service  utilizing  enhanced  security,  communication  and  collaboration  features  36.99 


The  results  shown  in  Table  4 (as  extracted  from  Figure  5)  indicate  a systematic  preference  for  education 
topics  related  to  efficient  administration  of  school  computer  labs  (ET4),  web  hosting  and  web  content 
publishing  using  well-established  CMS  software  (ETl 6 and  ETl 7),  best  practices  for  Internet  safety  (ET22) 
and  open  source  educational  applications  (ET15).  Eurthermore,  a considerable  preference  for  innovative 
educational  services  (ET20)  and  advanced  IT  devices  (ET12)  was  also  declared.  On  the  other  hand,  lower 
preference  was  recorded  for  potential  training  in  relative  obsolete  software  (ET6  and  ET7)  and  widely  used 
and  thoroughly  tested  services  provided  by  GSN,  such  as  the  email  service  (ET13).  Accordingly,  Eigure  4 
indicates  that  GSN  users  are  aware  of  the  advanced  electronic  services  offered  and  that  are  highly  utilized  in 
educational  purposes.  Also,  a noticeable  portion  of  the  participants  is  able  to  look  for  documentation  and 
official  guidelines  for  the  electronic  services  provided  by  the  GSN.  However,  a considerable  portion  of  the 
participants  is  unaware  where  can  efficiently  access  technical  documentation  for  school  computer 
laboratories  and  their  IT  infrastructure  and  also  not  properly  informed  considering  the  technical  support 
services  for  troubleshooting  IT  failures  at  school  PC  laboratories. 
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6.  CONCLUSIONS 

The  present  work  introduces  a time  Scheduling  and  tele-Education  Platform  (t-STEP)  based  on  the  Greek 
School  Network  (GSN)  and  defines  the  conceptual  framework  for  a technical-oriented  training.  A 
questionnaire  survey  was  performed  in  order  to  investigate  the  potential  technological  needs  of  computer 
science  educators  in  Greek  schools  (primary  and  secondary  education)  to  improve  their  technical  proficiency. 
Additionally,  typical  issues  considering  the  GSN  services  and  ICT  school  infrastructure  were  also  examined. 
Preliminary  results  indicate  a systematic  preference  trend  in  specific  administrative,  web -related  and 
advanced  IT  training  topics  and  furthermore  validate  the  broad  use  and  acceptance  of  GSN  services.  In  future 
work  a more  detailed  and  sophisticated  statistical  analysis  will  be  undertaken  in  order  to  formally  describe 
and  assess  the  impact  and  the  effectiveness  of  the  training  course  and  potentially  enhance  it. 
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ABSTRACT 

This  work  proposes  the  definition  of  an  Adaptive  Conversation-based  Learning  System  (ACLS)  able  to  foster  computer- 
mediated  tutorial  dialogues  at  the  workplace  in  order  to  increase  the  probability  to  generate  meaningful  learning  during 
conversations.  ACLS  provides  a virtual  assistant  selecting  the  best  partner  to  involve  in  the  conversation  and  generating 
adaptive  feedbacks  for  the  dialog.  Adaptive  feedbacks  are  triggered  by  the  concepts  automatically  extracted  from  the 
conversation  texts,  while  their  content  is  generated  by  querying  the  organizational  knowledge  represented  by  means  of 
Semantic  Web  technologies.  Lastly,  Fuzzy  Formal  Concept  Analysis  is  exploited  to  conceptualize  domain  knowledge. 


KEYWORDS 

Workplace  Learning,  Knowledge  Extraction,  Adaptive  Educational  System,  Semantic  Web,  Euzzy  Formal  Concept 
Analysis. 


1.  INTRODUCTION  AND  MOTIVATIONS 

Workplace  Learning  represents  the  field  of  studies  and  researches  related  to  effective  and  efficient  solutions 
supporting  learning  and  training  processes  within  the  work  context  and  aiming  at  enhancing  individual  and 
organization  performances. 

Workplace  Learning  principles  are  described  in  several  works.  Among  them,  the  authors  of  (M.  Wang,  et 
ak,  2010)  assert  that  Workplace  Learning  is  adult  learning,  organizational  learning  and  knowledge 
management.  The  related  theories  emphasize  personal  reflection,  problem  orientation  and  knowledge 
construction  by  means  of  social  processes,  models  representing  how  organizations  learn,  approaches  and 
practices  exploited  in  order  to  identify,  create,  represent,  and  distribute  knowledge  for  reuse,  awareness  and 
learning.  Furthermore,  in  (P.  Tynjala  and  P.  Hakkinen,  2005),  the  authors  describe  the  main  features  of  the 
Workplace  Learning.  First  of  all,  it  is  mostly  informal  or  non-formal  (both  intentional  and  incidental). 
Secondly,  it  is  strongly  contextualized  in  the  sense  that  learning  occurs  in  the  environment  in  which  skills  and 
knowledge,  that  are  object  of  learning  itself,  will  be  applied.  In  this  scenario,  conversation  is  considered  an 
important  tool  to  share,  construct,  create  knowledge  and  learn  as  emphasized  in  (A.  Soller,  2007).  The 
authors  of  (I.  Nonaka  and  H.  Takeuchi,  1995)  underline  the  importance  of  conversations  in  order  to  transform 
individual  processes  into  organizational  processes. 

Moreover,  conversations  foster  personal  reflection  and  are  typically  driven  by  a well-defined  learning 
objective.  In  this  context,  technology  enhanced  learning  solutions  are  effective  not  only  to  support 
conversations  (dialogues,  discussions,  etc.)  but  also  to  store  knowledge,  ideas  and  shared  decisions.  They  can 
serve,  at  the  same  time,  as  a tool  to  support  individual  learning,  sustain  knowledge  creation  and  construction, 
manage  the  organizational  memory,  share  knowledge  and  develop  mutual  understanding  (M.  Wang  et  ak, 
2010).  Generally,  these  systems  are  able  to  catch  the  knowledge  of  the  domain  experts  to  support  learning 
and  peer  collaboration  but  they  need  methodical  approaches  for  knowledge  representation  (W.  Chen,  2006), 
otherwise  they  are  focused,  as  Adaptive  and  Intelligent  Systems  for  Collaborative  Learning  Support  (AICLS), 
on  adapting  the  collaboration  processes  (I.  Magnisalis  et  ak,  201 1). 

With  respect  to  the  aforementioned  systems,  this  paper  proposes  a novel  approach  to  exploit 
organizational  resources  (kept  up-to-date  by  the  collective  intelligence  and  represented  by  means  of  semantic 
models)  in  order  to  enhance  and  adapt  peer  learning  activities. 
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Taking  in  consideration  the  relevant  role  of  conversation  at  workplace  for  both  individual  and 
organizational  learning  and  for  knowledge  management,  this  paper  proposes  a workplace  learning  system, 
based  on  semantic  technologies,  that  implements  the  conversation-based  learning  approach.  The  main  faced 
problems  are:  i)  empowering  conversations  in  order  to  facilitate  the  occurrence  of  learning  by  providing  a 
mechanism  to  stimulate  meaningful  learning,  and  ii)  exploiting  conversations  as  a tool  to  link  individual  and 
organizational  learning  by  tracing  and  reusing  learningful  conversations. 


Virtual  Assistant  Learner 

Q ^ 


Tutor  Virtual  Assistant 

Q 


Learner  Tutor 

^ & 


^■'aboratior 


concept  extraction: 
"Semantic  Web" 


TEXT 

<fragment> 


<feedback> 

This  conversation  is 
related  to  the  Project  X 
dealing  v^ith  FOAF  you 
have  been  involved  in 


<message> 
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between  Semantic  Web  and 
FOAF?“ 


<message> 
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1 


"FOAF  is  vocabulary 
exploiting  Semantic  Web 
technologies  and  it  is 
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ieiaboration- 


concept  extraction: 
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<messaqe> 

<messaqe> 

/' 

<messaqe> 
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<message> 

<fragment> 

"Let  us  go  further  in 

applications  of  the 
Semantic  Web  ...  we  can 
use  Document  Y ... " 

<feedback> 

<message> 

dealing  with  the  use  of 

<message> 

Search  Engine 
performances 

"Ok.  Now  the  advantages 
of  Semantic  Web  are 
clear  for  me" 


(a)  learner-related  feedback 


(b)  tutor-related  feedback 


Figure  1 . Learner-related  and  tutor-related  feedback 

The  structure  of  this  work  is  the  following;  section  II  provides  an  overview  of  the  defined  approach  and  a 
description  of  the  adaptation  strategy.  Section  III  describes  the  general  view  and  the  most  important  details  of 
the  ACTS.  Lastly,  Section  IV  provides  details  about  the  evaluation  of  the  proposed  approach  and  some  final 
remarks. 


2.  OVERALL  APPROACH 

The  proposed  approach  lays  on  three  pillars  mainly  enabling  a virtual  assistant  that  exploits  the 
organizational  knowledge  in  order  to  foster  conversations  by  means  of  the  provision  of  feedbacks  for  both 
learner  and  conversation  partner. 

The  first  pillar  is  represented  by  computer-mediated  conversations.  In  our  approach,  conversations  are 
dialogues  between  two  participants,  the  tutor  and  the  learner,  who  exchange  messages  through  an  instant 
messaging  tools.  A model  for  learningful  conversations  is  defined  in  (D.  Laurillard,  2009),  where  the  author 
provides  a framework  for  a conversational  learning  approach.  This  framework  is  conceptually  structured  into 
two  levels,  the  lower  and  the  upper.  In  the  lower  one,  the  learner  masters  the  topics  of  learning  while  the 
conversation  partner  provides  the  experiential  environment  (e.g.  delivery  of  learning  resources)  where  the 
learning  process  is  executed.  In  the  upper  level,  the  learner  and  the  conversation  partner  are  engaged  in  a 
dialogue  by  exchanging  messages  containing  their  understanding  and  representations  of  the  topics  obtained 
through  the  experience  performed  at  the  lower  level  and  adapting  their  behaviours.  Reflection  occurs  when 
the  learner  and  the  partner  talk  about  what  they  are  doing  at  the  lower  level.  Adaptation  occurs  when  they 
modify  what  they  are  doing  at  the  lower  label  on  the  basis  of  their  talk.  Several  types  of  dialogues  (e.g. 
argumentation-based  dialogues,  tutoring  dialogues,  peer  dialogues,  and  so  on)  can  be  instantiated,  but  this 
work,  in  particular,  focuses  on  tutoring  dialogues.  The  virtual  assistant  is  committed  to  help  the  conversation 
partner  in  playing  his/her  tutor  role. 
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The  second  pillar  is  represented  by  the  capability  to  generate  adaptive  feedbacks  able  to  foster 
conversations  in  order  to  increase  the  probability  that  meaningful  learning  occurs  during  dialogues.  We  adopt 
the  definition  of  feedback  reported  in  (V.  J.  Shute,  2008);  [...]  feedback  is  defined  in  this  review  as 
information  communicated  to  the  learner  that  is  intended  to  modify  his  or  her  thinking  or  behavior  for  the 
purpose  of  improving  learning.  And  although  the  teacher  may  also  receive  student-related  information  and 
use  it  as  the  basis  for  altering  instruction  [...].  In  our  proposal,  feedbacks  are  generated  by  the  virtual 
assistant  that  analyses  a specific  conversation  fragment  and  queries  the  organizational  knowledge  for  content 
that  could  foster  the  dialogue  and  help  the  learner  to  improve  development  of  domain-specific  knowledge 
and  skills.  Feedbacks  are  adaptive,  in  the  sense  that  they  are  generated  by  considering  the  concepts  that  really 
emerge  from  the  conversation  fragment  and  personalized,  in  the  sense  that  they  take  care  of  both  learner  and 
tutor  roles  and  are  tailored  to  prior  knowledge  and  previous  work  experience  of  the  learner.  For  the  sake  of 
simplicity,  we  divide  feedbacks  in  two  types:  learner-related  and  tutor-related.  The  first  ones  are  topic 
contingent  feedbacks  suggesting  correlations  among  the  topics  to  master  and  the  learner’s  prior  knowledge 
(V.  J.  Shute,  2008).  The  second  ones  are  hints/cues/prompts  about  worked  examples  provided  by  the  tutor 
(conversation  partner)  in  response  to  automatic  suggestions,  produced  by  the  system,  concerning  the 
existence  (in  the  organizational  knowledge)  of  documents,  user -generated  content,  etc.  that  are  related  to  the 
topics  to  master.  The  main  idea  is  to  build  these  feedbacks  in  the  form  of  dialogue  moves  by  exploiting 
classifications  provided  by  the  authors  of  (S.  D’Mello  et  al.,  2010)  and  (X.  Lu  et  al.,  2007).  In  this  way,  even 
if  indirectly,  the  dialogue  is  adapted  maintaining  a common  tutorial  dialogue  scheme. 


Learner 


Tutor 


^ ^ Laurillard's  Framework  Upper  Level 

^ ^ ^ Laurillard's  Framework  Lower  Level 


Figure  2.  Learner-related  and  tutor-related  feedback 

The  third  pillar  is  represented  by  the  exploitation  of  the  organizational  knowledge  in  order  to  support 
computer-mediated  conversations  as  well  as  other  processes.  In  this  paper  we  refer  to  the  organizational 
knowledge  as  the  set  of  all  types  of  knowledge  existing  in  a specific  organization,  for  instance,  tacit 
knowledge  in  the  minds  of  workers,  embedded  knowledge  in  procedures,  explicit  knowledge  recorded  in 
artefacts  (e.g.  documents,  etc.)  and  in  information  systems  (e.g.  information  about  the  competences  of  each 
workers  (N.  Capuano  et  al.,  2011),  etc.),  and  so  on.  In  our  approach  the  organizational  knowledge  is 
represented  by  means  of  a model  (see  section  III-A  for  further  details)  exploiting  the  Semantic  Web  stack*. 

The  so  represented  organizational  knowledge  is  mainly  useful  to  accomplish  three  objectives.  The  first 
one  is  to  enable  search  for  suitable  conversation  partners  among  all  the  available  human  resources  in  the 
organization.  The  second  one  is  to  enable  search  for  resources  (e.g.  documents,  user  generated  content,  task 


' http://www.w3.org/2001/sw/ 
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and  project  information,  and  so  on)  useful  to  generate  personalized  and  adaptive  feedbacks  fostering 
learningful  conversations  (in  this  case  the  virtual  assistant,  once  extracted  the  concepts  from  the  conversation 
fragment,  uses  SPARQLl.l^  to  query  the  organizational  knowledge  and  provide  content  to  construct 
feedbacks).  The  third  one  is  to  enable  storage  and  correlation  of  learningful  conversations  with  the  existing 
knowledge  in  organization  in  order  to  foster  reuse.  For  a further  description  of  the  overall  approach.  Fig.  1 
shows  the  two  types  of  feedbacks  and  how  they  support  adaptation  of  the  conversation  by  providing 
suggestions  to  the  learner  (Fig.  la)  and  to  the  tutor  (Fig.  lb). 


3.  ADAPTIVE  CONVERSATION-BASED  LEARNING  SYSTEM 

Our  ACTS  implements  the  approach  described  in  II.  The  high  level  architecture  of  it  is  presented  in  fig.  2. 

3.1  Structuring  the  Organizational  Knowledge 

Modelling  and  representing  the  organizational  knowledge  are  two  of  the  most  important  tasks  related  to  the 
definition  of  the  ACTS  architecture. 

In  particular,  the  technologies  adopted  to  represent  the  organizational  knowledge  come  directly  from  the 
W3C  Semantic  Web  vision.  This  choice  guarantees  a layer  of  interoperability  and  cooperation  among 
applications  (or  apps),  the  fundamentals  to  build  knowledge-based  applications,  the  chance  to  use  a standard 
query  language  like  SPARQLl.l,  the  possibility  to  integrate  and  reuse  existing  ontologies,  vocabularies  and 
metadata  to  model  several  aspects  of  the  organisational  knowledge,  the  capability  to  support  reasoning, 
inference  and  so  on. 

If  the  Semantic  Web  provides  us  with  a set  of  methodologies,  languages  and  technologies  useful  to 
represent  the  organizational  knowledge,  an  effective  and  efficient  organizational  knowledge  model  is  needed. 
The  solution  for  the  aforementioned  issue  is  provided  by  the  ARISTOTELE  Project^,  where  the 
organizational  knowledge  is  presented  as  Organization  Linked  Data  structured  in  three  layers  as  depicted  in  3 
that  provides  only  a fragment  of  the  whole  model.  Eirstly,  the  upper  layer  consists  of  several  linked 
ontologies  (described  by  using  RDES/OWL/OWL2"')  used  to  model  the  organization  key  concepts  (ontology 
classes).  Secondly,  the  lower  layer  consists  of  the  instances  of  the  classes  we  can  find  in  the  upper  layer. 
Lastly,  the  middle  layer  is  made  of  a set  of  lightweight  ontologies  used  to  classify  and  organize  the  lower 
layer  elements.  Lightweight  ontologies  (described  by  using  SKOS^)  can  be  connected  each  other  in  order  to 
correlate  concepts  (at  the  same  layer)  and  instances  (at  the  lower  level). 

More  in  details,  the  ontologies  at  the  upper  layer  describes  the  semantics  of  domain-independent  concepts 
in  organization  like  Task,  Competence,  Worker,  Content,  Document,  BlogPost,  etc.  that  are  implemented  as 
OWL  classes.  Whilst,  the  middle  layer  defines  conceptualizations  for  domain-dependent  knowledge  in  a 
specific  organization.  Lor  instance,  the  main  research  topics  the  organization  deals  with  are  modelled  as 
instances  of  skos  : concept  and  organized  in  semantic  structures  like  taxonomies  or  conceptual  maps. 

It  is  clear  that  the  middle  layer  is  more  dynamic  than  the  upper  layer,  in  the  sense  that  the  lightweight 
ontologies  (as  we  have  defined  them)  can  evolve  in  the  time  if,  for  instance,  a new  research  field  is  activated 
or  new  project  artefacts  are  indexed  in  the  Document  Management  System  (DMS)  of  the  organization. 
Instead,  the  probability  that  a concept  (like,  for  example.  Document  or  Task)  changes  in  the  upper  layer  is 
very  low.  The  construction  of  the  lightweight  ontologies  implementing  the  middle  layer  is  a critical  and 
difficult  task. 

The  idea  is  to  generate  the  aforementioned  ontologies  by  exploiting  textual  data  embedded  in  documents 
(M.  Gaeta  et  al.,  2011)  that  are  representative  for  the  organizational  knowledge.  Lor  this  aim  we  exploit  the 
framework  described  in  (C.  De  Maio  et  al.,  2012)  that  is  based  on  a fuzzy  extension  of  the  Lormal  Concept 
Analysis  (Q.  T.  Tho  et  al.,  2006).  The  objective  of  the  above  mentioned  framework  is  building  a taxonomical 
conceptual  structure  starting  from  a collection  of  text  documents.  The  framework  defines  an  ontology 
generation  workflow  consisting  of  three  main  steps:  text  processing,  fuzzy  data  analysis  and  ontology 
building. 


^ http://www.w3.org/TR/sparqll  l-query/ 

^ http://www.aristotele-ip.eu/en/project/svc/entry/0/196/4/aristoteleproject.htral 
http://www.w3.org/TR/owl2-overview/ 

^ http://www.w3.org/2004/02/skos/ 
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Figure  3.  Three  layers  of  structured  Organisational  Knowledge  (Organisational  Linked  Data) 

The  goal  of  the  first  step  is  to  construct  a Fuzzy  Formal  Context,  i.e.  a matrix  showing  the  relationships 
between  the  keywords  extracted  from  the  input  documents  and  the  documents. 

The  set  of  keywords  is  extracted  from  the  documents,  filtered  (by  eliminating  non-informative  words  by 
using  stopword  lists),  normalized  (by  means  of  stemming  and  POS  tagging)  and  enriched  by  inserting  the 
synonyms  of  all  words  in  the  set.  The  relationship  value  (in  the  range  [0,7])  in  a matrix  cell  (g,m)  (g  is  one  of 
the  input  documents  and  m one  of  the  keyword  in  the  enriched  set)  is  calculated  by  using  the  TD-IDF 
technique  (term  frequency  inverse  document  frequency)  and  represents  an  evaluation  of  the  measure  of 
strength  of  the  relationship.  The  goal  of  the  second  step  is  to  analyse  the  Fuzzy  Formal  Context  by  means  of 
Fuzzy  Formal  Concept  Analysis  (FFCA)  and  transform  the  matrix  into  a Fuzzy  Concept  Lattice  (nodes  in  the 
lattice  are  called  Fuzzy  Formal  Concepts)  that  is  a mathematical  modelling  of  the  knowledge  embedded  in 
the  input  documents  and,  moreover,  it  can  be  considered  an  alternative  and  more  informative  representation 
of  the  matrix.  Lastly,  the  goal  of  the  third  step  is  to  transform  the  Fuzzy  Concept  Lattice  into  a taxonomy 
structure  by  executing  some  rules.  In  our  approach  we  use  a SKOS -based  representation  of  the  final 
taxonomy  instead  of  the  OWL-based  representation  adopted  in  (C.  De  Maio  et  al.,  2012).  SKOS  is  more 
suitable  than  OWL  when  the  objective  is  to  organize  large  collections  of  objects  and  provide  a lightweight 
intuitive  conceptual  modelling.  At  the  end  of  the  process,  the  obtained  SKOS  structures  represent  the 
aforementioned  lightweight  ontologies.  It’s  important  to  underline  that  the  documents,  used  as  input  of  the 
ontology  construction  process,  are  already  related  to  their  respective  concepts  in  the  SKOS  structures. 

New  documents,  as  well  as  other  artefacts,  can  be  subsequently  classified  (by  manual  and/or  automatic 
operations)  by  the  lightweight  ontologies. 

Definitely,  the  lightweight  ontologies  can  evolve  by  exploiting  a similar  process  based  again  on  FFCA. 

3.2  Activation  of  a Context-Steered  Conversation  and  Selection  of  a Suitable 
Tutor 

First  of  all,  it  is  important  to  remark  the  role  of  the  context  in  which  the  conversation  is  activated.  The 
learning  scenario  we  consider  in  this  work  is  the  context-steered  learning  (A.  Schmidt  and  S.  Braun,  2006) 
where  a worker  has  been  committed  to  execute  a specific  task  (an  instance  of  the  Task  class)  that  requires  a 
specific  competence  (an  instance  of  the  Competence  class)  that  must  be  developed  (fully  or  partially)  by 
means  of  the  execution  of  a tutorial  conversation.  Now,  the  context  is  defined  as  the  set  of  all  concepts 
(nodes  of  the  lightweight  ontologies)  that  are  directly  linked  to  the  needed  competence  or  to  its  parts 
(knowledge,  skills  or  attitudes). 
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The  so  defined  context  is  useful  to  select  a suitable  conversation  partner.  This  selection  is  one  of  the  most 
important  operations  enabled  by  the  Organizational  Linked  Data  and  is  realized  by  using  SPARQL  queries. 
In  order  to  find  suitable  partners  (peers,  experts,  tutors)  among  all  the  workers  in  the  organization  our 
approach  takes  into  account,  their  competences  (see  classes  Competence,  Knowledge,  Skill  and  Attitude), 
work  experiences  (see  Task  class)  and  produced  artefacts  (see  Content,  Document  and  BlogPost  classes). 

For  instance,  it  is  possible  to  write  a query  to  find  all  the  workers  with  Software  Engineering  and 
Tutoring  competences  in  order  to  reinforce  the  tutor  role.  Moreover,  it  is  also  possible  to  relax  the  constraint 
on  Tutoring  competence  and  search  only  for  Software  Engineering  competence  to  have  a peer-based 
conversation.  With  respect  to  the  architecture  presented  in  Fig.  2,  the  module  responsible  for  finding  a 
suitable  tutor  is  the  Tutor  Selector. 

3.3  Feedback  Generation 

Once  the  lightweight  ontologies  are  generated  and  deployed,  we  have  two  types  of  elements  linked  to  them: 
the  set  D{c)  of  all  documents  and  the  set  E(c)  of  all  elements  that  are  instances  of  classes  Content  (or  its 
subclasses)  and  Task.  The  main  idea  is  using  a search  engine  like  Lucene^  to  index  all  the  elements  in  the  set 
D(c)  and  finding  matches  among  a conversation  fragment  T (extracted  by  means  of  the  Instant  Messaging 
Tool  presented  in  Fig.  2 by  exploiting  the  MoreLikeThis  function  provided  by  the  Lucene  API). 


The  virtual  assistant  invites  the  tutor  to  provide  prompts,  hints  or  cues  by  using  worked  examples 
retrieved  from  the  documents,  blog  posts,  wiki  articles,  task/project  information  and  so  on.  The  idea  is  that 
the  tutor  has  the  competence  to  mediate  the  aforementioned  artefacts  and  use  them  as  learning  content. 
Furthermore,  the  virtual  assistant  suggests  the  learner  to  reflect  on  (or  to  ask  the  tutor  for  details  about)  the 
correlation  among  his/her  previous  work  experience,  represented  by  the  retrieved  elements  and  his/her 
understanding  of  the  current  conversation  fragment. 

With  respect  to  the  architecture  presented  in  Fig.  2,  the  module  responsible  for  finding  a suitable  tutor  is 
the  Personalized  and  Adaptive  Feedback  Generator.  Feedbacks  for  the  learner  and  the  tutor  are  provided 
by  means,  respectively,  of  the  Learner’s  Viewer  of  Feedback  and  the  Tutor’s  Viewer  of  Feedback  shown 
in  the  high-level  architecture.  Table  1 provides  a mapping  between  the  generated  feedbacks  and  the 
interaction  patterns  in  tutorial  dialogues  (X.  Lu  et  al.,  2007)  (S.  D’Mello  et  al.,  2010). 


* http://lucene.apache.org/core/ 


I I indexed  documents  O lightweight  ontology  concepts  ■ other 


Other  resources  classified 
with  ontology  concepts 


Figure  4.  Matching  example 
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Table  1 . Mapping  feedbacks  on  dialogue  patterns 


Feedback 

Pattern 

Content 

Suggesting  personal  refieelion 
on  integration  among  conversation 
topics  and  learn|er’s  prior  knowledge. 

Student-tutor. 

Set  .4. 

Suggesting  quesTions  (to  the  tutor) 
about  integration  among  conversation 
topic  and  learners'  prior  knowledge. 

Student-tutor. 

Set  A. 

Suggesting  specific  prompting 
by  instructing  with  worked  examples 
coming  from  real  and  concrete 
work  experiences. 

Tutor-student. 

Set  B. 

3.4  Knowledge  Reuse 

In  ACLS,  conversation  threads  are  traced  and  indexed  to  satisfy  a possible  need  to  reuse  them  in  informal  or 
non-formal  learning  experience.  In  order  to  foster  reuse,  conversation  threads  are  represented  by  using 
SIOC^.  SIOC  may  be  easily  integrated  with  the  upper  layer  ontologies  of  the  Organizational  Linked  Data 
because  they  share  the  same  Semantic  Web  stack.  In  particular,  we  use  some  extensions  of  SIOC  (e.g. 
sioct:ChatChannel  and  sioct: InstantMessage  that  are  sub-classes  of  sioc:Forum  and  sioc:Post)  to  model  a 
conversation  session  and  individual  messages.  With  respect  to  our  work,  the  most  important  properties  of  the 
sioct:InstantMessage  class  are  topic  and  content. 

The  first  one  enables  to  link  a conversation  message  with  a SKOS:  Concept.  This  is,  de  facto,  a way  to 
index  messages  and  threads  by  means  of  lightweight  ontologies  at  the  middle  layer  of  the  Organizational 
Linked  Data.  The  second  one  stores  textual  data  of  a message.  Thus,  a Social  Semantic  Web  process  is 
deployed;  conversations  produces  messages  and  threads  represented  in  SIOC  that  are  classified  with  respect 
to  the  concepts  in  the  lightweight  ontologies  and  become  retrievable  through  SPARQL  queries.  The  quality 
of  the  conversation  messages  and  threads  can  be  evaluated  by  means  of  social  rating  or  by  assessing  learners 
after  the  conversations. 


4.  EVALUATION  AND  FINAL  REMARKS 


In  order  to  emphasize  the  rationale  of  the  feedback  generated  by  the  ACLS  with  respect  to  the  objective  of 
improving  the  meaningfullness  of  learning  during  tutorial  dialogues.  Fig.  5 provides  the  rules  used  to  define 
the  feedbacks. 


Table  2.  Rules  used  to  generate  the  feedbacks  and  their  rationale 


Connecting  learner's  prior  People  learn  better  when  unfamiliar 

1 knowledge  with  new  one  material  is  related  to  familiar  knowledge 


and  when  they  ask  questions 

Connecting  learner's  prior  People  learn  better  when  they  organize 

2 experience  with  new  knowledge  and  connect  new  concepts  with  already 

acquired  ones 

Enriching  explanation  with  People  learn  better  when  worked 

3 expert's  concrete  work  examples  are  presented  in  the  context  of 

experiences  a familiar  situation 

^ Exploiting  organisational  People  learn  better  with  guidance  rather 

resources  as  learning  content  than  by  pure  discovery 


In  brief,  according  to  (R.  E.  Mayer,  2008),  the  generated  feedbacks  try  to  stimulate  generative  processes 
(e.g.  organizing  and  integrating  knowledge)  which  are  those,  among  the  cognitive  processes,  able  to  produce 
meaningful  learning.  Furthermore,  generative  processes  are  sustained  also  by  the  defined  Organizational 
Linked  Data  in  the  sense  that  the  Semantic  Web  structures,  and  in  particular  the  explicit  use  of  the 
lightweight  ontologies,  help  the  learner  to  organize  the  knowledge  and  integrate  the  new  one  with  the  prior 


’ http://rdfs.org/sioc/spec/ 
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one.  Unlike  what  the  other  systems  usually  do,  the  proposed  approach  pays  attention  to  both  the  domain 
knowledge  and  the  cognitive  processes  raised  from  the  partners’  interactions  but  it  doesn’t  need  any 
expensive  modelling  activity.  Both  the  approach  and  the  ACLS  will  be  experimented  in  the  next  months  in 
the  ARISTOTELE  Project  activities. 
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ABSTRACT 

Individual  learning  is  out  of  sync  with  the  elements  of  a curricula  and  the  daily  program  of  a teacher.  At  a time  when 
multidigit  multiplication  methods  are  taught,  many  children  are  not  perfectly  performing  the  basic  multiplication  table. 
Teachers  organize  settings  for  learning  and  they  usually  have  no  time  to  give  an  individual  feedback  to  every  single 
student.  Furthermore  they  have  no  physiological  capacity  to  produce  statistics  and  to  investigate  the  probable  causes  of 
errors.  Our  goal  is  to  give  teachers  a tool  to  help  them  to  review  and  train  some  typical  requirements  for  written 
multiplying.  We  use  the  ubiquity  of  the  Internet  and  provide  on  different  devices  free  applications  with  written 
multiplication  problems.  All  data  are  collected  centrally  and  are  used  to  determine  typical  errors.  These  data  could  also  be 
used  to  answer  further  new  questions  and  test  new  hypothesis.  This  is  one  aspect  of  Learning  Analytics.  The  given  tasks 
are  automatically  adapted  to  the  skill  level  of  a student.  Teachers  can  see  every  detail  of  the  solution  process  of  every 
task.  But  really  useful  is  that  teachers  can  see  the  typical  problems  of  each  student  as  well  as  of  the  whole  class  every 
time  at  a glance.  They  need  just  a connection  to  the  database  with  a web  browser  - anywhere,  anytime. 


KEYWORDS 

Written  multiplication,  feedback,  learning  analytics,  degree  of  competence,  algorithm,  school  children 


1.  INTRODUCTION 

Even  before  the  advent  of  the  first  personal  computers  (PC)  in  classrooms,  educational  researchers  attempted 
to  realize  an  intelligent  tutoring  system  (ITS)  [13].  Early  attempts  to  model  student  knowledge  were  based  on 
a "failure"  model  for  subtraction  with  more  than  100  typical  problems  described  by  Brown  and  Vanlehn  [2]. 
This  is  based  on  the  idea  that  computers  can  “perceive”  and  evaluate  much  faster  and  more  detailed 
information  about  some  aspects  of  the  behavior  and  the  learning  process  of  each  individual  student  in  a class 
as  a teacher.  Even  teachers  have  always  too  little  time.  Therefore  in  the  past  the  extensive  listings  of  results 
and  protocols  have  been  ignored  or  rejected  by  the  teachers.  The  PC  with  restricted  capacity  brought  some 
simple  drill  & practice  programs  into  the  classrooms.  Eor  our  subject  “multidigit  multiplication”  there  are 
actual  applications  referred  in  [1]  [9]  [11]  for  example.  Computer  power  today  is  often  used  for  the  delivery 
of  content  and  for  the  administration.  Adaptive  skill  tests  are  often  used  to  efficiently  estimate  a global 
performance  value. 

We  at  the  TU  Graz  are  developing  some  applications  (little  helpers)  which  realize  the  idea  to  check  up 
and  train  only  a special  curricular  area.  This  is  based  on  the  experience  that  teachers  refuse  any  innovations 
when  they  tend  to  intervene  heavily  in  their  management  of  teaching.  May  be  that  is  a reason  that  it  seems 
that  the  ITS-tradition  has  been  forgotten  in  classrooms.  Today,  however,  we  see  more  and  more  a 
confrontation  between  the  general  requirements  for  greater  individualization,  for  more  and  more  precise 
feedback  [8],  and  this  denial  of  the  teacher  towards  the  support  of  technology. 

We  agree  with  Phil  Long  and  George  Siemens  who  stated  [10]  that  “the  most  dramatic  factor  shaping  the 
future  of  (higher)  education  is  something  that  we  can’t  actually  touch  or  see:  big  data  and  analytics”,  the 
founders  of  the  research  field  Learning  Analytics.  And  on  of  the  main  issues  is  the  question  what  exactly 
must  be  measured  to  get  a deeper  understanding  of  learning  processes  [4].  Eurthermore  nowadays  a 
technological  shift  is  happening  to  more  flexible  and  ubiquitous  applications.  They  are  to  be  used  in  browsers 
or  even  as  applications  on  mobile  devices.  This  new  technological  environment  and  a new  understanding  of 
Learning  Analytics  at  Graz  University  of  Technology  (TU  Graz)  are  realized  in  applications  that  can  be  used 
by  students  on  many  occasions  and  that  could  contribute  to  an  automation  of  basic  mathematical  operations 
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[12].  Basic  multiplication  facts  based  on  an  automatic  level  of  proficiency  are  considered  to  be  foundational 
for  further  progress  in  mathematics  [7].  Using  this  software  the  arithmetic  technique  should  already  be 
practiced  in  traditional  ways.  The  program  is  not  primarily  intended  to  use  to  teach.  It  is  a diagnostic  and 
training  tool  and  gives  feedback  about  typical  problems  in  the  domain.  It  can  be  used  with  no  further 
guidance  or  explanation.  The  program  is  intended  to  relieve  the  teacher,  who  usually  does  not  have  enough 
time  and  concentration  to  look  very  carefully  for  the  causes  of  the  mistakes.  We  had  a look  to  the  analog 
episode  in  the  program  TutorIT  [11]  and  we  interpret  its  intrusive  help  in  small  steps  rather  as  a blockade 
than  an  encouragement.  Our  applications  checks  in  particular,  whether  the  solution  is  correct,  and  then  we  try 
to  investigate  the  approach.  Of  course,  if  we  get  only  an  incorrect  result,  no  carries,  no  partial  calculations, 
we  cannot  go  on  with  an  analysis. 

The  main  idea  of  our  approach  is  to  collect  every  student’s  data  in  a central  database.  Tasks  are  given  by  a 
browser  based  program  and  by  apps  for  iOS  and  Android.  If  a student  has  completed  a multidigit 
multiplication,  the  program  analyzes  different  aspects.  For  example  if  the  solution  is  not  correct,  the  program 
looks  for  evidence  on  typical  failures.  Is  there  a problem  with  basics,  with  simple  multiplication  facts?  Is 
there  a lack  of  overall  competence?  Are  there  problems  with  the  correct  placement  and  alignment  of  the 
carries  or  when  writing  down  the  partial  results?  Many  individual  data  are  compressed  into  one  compact 
statement.  This  application  can  give  teachers  a powerful  hint  for  the  further  promotion  of  each  individual 
student.  We  know,  that  “the  mathematicians  were  variable  and  versatile  in  their  strategy  use,  as  shown  most 
strikingly  by  the  fact  that  when  18  of  the  mathematicians  were  retested  after  a few  months,  they  used 
different  strategies  on  the  two  occasions  for  most  of  the  same  problems’’  [3,  p.20].  Considering  this  aspect  the 
correct  solution  must  be  recognized  in  any  case  and  learners  should  not  be  forced  to  specific  solution  steps. 

Finally,  it  is  a major  goal,  that  the  results  and  the  evaluation  should  not  overburden  teachers  with  details. 
Based  on  the  collected  data  it  is  possible  to  generate  detailed  analyses,  almost  broken  down  to  the  single 
multiplications  which  has  been  shown  to  solve  the  task.  But  we  said  it  already,  it  must  be  taken  into  account 
that  teachers  have  only  a limited  time  budget  for  their  interpretation.  Therefore  normally  the  teachers  were 
“alarmed”  for  doing  some  urgent  interventions  only  in  a few  cases.  It  can  be  summarized  that  our  approach  is 
to  develop  a small  web-based  application  where  each  single  pupil  does  not  get  only  a random  selection  of 
multidigit  multiplication  tasks.  The  program  estimates  a competence  level  and  generates  a hierarchy  of 
adequate  new  problems.  The  teacher  can  see  just  in  time  the  results  and  analysis  of  each  single  task  just  after 
the  student  has  finished  it. 


2.  GENERAL  CONCEPT 

First  of  all,  the  program  targets  school  children  in  classes  4 and  5 - starting  from  an  age  of  about  10  years. 
The  name  and  layout  of  the  web-based  program  was  chosen  with  regard  to  children:  easy  and  playful;  it  is 
called  “Multi-Math-Coach”  Learning  by  playing  is  an  important  concept  in  this  respect.  Knowledge  of  the 
basic  multiplication  tables  and  the  ability  of  working  with  an  Internet  browser  are  the  prerequisites  for  using 
the  learning  application.  It  is  suggested  that  teachers  should  always  give  an  initial  introduction  and  overview 
to  children  of  how  to  use  and  deal  with  the  program  before  using  it.  For  displaying  multiplications  in  the 
front-end,  the  standard  school  method  for  written  multiplications  is  used  [5]. 

The  crucial  goals  of  our  application  are: 

1.  An  overall  learning  algorithm  that  sets  the  difficulty  for  multiplications  adaptively  and 
independently  for  each  user, 

2.  The  generation  of  multiplications  of  a specific  category  (of  difficulty), 

3.  The  analysis/evaluation  of  the  results  of  the  generated  and  displayed  multiplications  (=  user 
inputs),  typification  of  the  detailed  results  (multiplication,  addition  alignment). 

It  has  already  been  mentioned  that  the  learning  application  should  adapt  to  the  current  user’s  expertise  in 
multidigit  multiplication  knowledge.  To  do  so,  in  the  very  first  beginning  categories  of  multiplications  that 
differ  in  difficulty  must  be  defined.  Another  main  objective  of  the  multiplication  generation  algorithm  is  to 
stay  universal  and  to  offer  a large  variety  of  examples  at  the  same  time.  On  the  basis  of  [6]  eight  different 
problem  groups  of  multidigit  multiplication  are  distinguished.  The  overall  domain  of  definition  includes 
multiplicands  (left  factor)  with  2 to  4 digits  and  multipliers  (right  factor)  starting  from  1 up  to  3 digits  in  size. 
All  combinations  of  such  factors  result  in  a total  of  eight  problem  groups,  which  are  listed  in  Table  1.  A 
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specific  problem  group  only  depends  on  the  size  of  its  both  factors.  Problem  groups  1 to  3,  the  simplest 
categories  of  multidigit  multiplication,  are  summed  up  to  so-called  Expertise  category  I which  is  formed  of 
multipliers  with  only  one  digit  in  size  whereas  the  multiplicand  varies  in  size.  As  can  be  seen  in  Table  1,  a 
total  of  three  overall  Expertise  categories  are  classified  like  this. 

Table  1.  The  domain  of  definition  for  the  Multi-Math-Coach. 


Expertise 

Group 

Pattern 

I 

1 

nn  * n 

2 

nnn  * n 

3 

nnnn  * n 

II 

4 

nn  * nn 

5 

nnn  * nn 

6 

nnnn  * nn 

III 

7 

nnn  * nnn 

8 

nnnn  * nnn 

The  outcome  of  Table  1 now  lets  us  differentiate  between  three  universal  categories  of  examples,  each 
including  a more  fine-grained  classification  inside  each  Expertise  class,  which  allows  a universal  allocation 
of  multiplications  in  difficulty.  Beside  these  categories  of  difficulty  the  concept  of  written  addition  and 
multiplication  includes  another  dimension:  the  carry.  In  our  application  simple  multiplications  without  carries 
and  multiplications  including  a carry  are  separated.  As  a whole  the  program’s  domain  consists  of  eight 
problem  groups  for  simple  multiplications  and  the  same  eight  problem  groups  for  multiplications  including 
one  or  more  carries.  The  ability  of  the  application  to  adapt  to  the  particular  skills  of  the  students  is  based  on 
generating  or  providing  appropriate  test  items  according  to  this  16  hierarchy  levels. 

2.1  Generation  of  Multiplications 

The  algorithm  for  generating  multiplications  on  the  described  sixteen  levels  differs  from  problem  group  to 
problem  group.  After  a couple  of  serious  tests  it  was  realized  that  much  of  the  thinking  and  strategies  for 
generating  the  factors  randomly  was  not  effective.  The  simulations  showed  that  factors  with  the  digits  1 and  0 
are  more  frequent  than  expected.  Eor  the  first  concept  (tasks  with  no  carry)  it  is  very  difficult  to  build 
multidigit  multiplications. 

One  pre-condition  for  the  algorithm  is  the  absence  of  identical  digits  in  a factor.  Eurthermore  factors  must 
be  generated  under  the  condition  that  no  carry  is  needed  to  solve  the  multiplication. 

A bunch  of  test  runs  carried  out  specific  frequencies  for  the  occurrence  of  specific  digits  for  particular 
problem  groups.  Larger  numbers  in  the  multiplicand  require  smaller  numbers  at  the  multiplier,  if  a carry  must 
be  avoided.  See  Table  2 as  an  example  where  the  probabilities  for  digits  for  multipliers  of  (the  simplest) 
problem  group  1 are  shown  for  multiplications  without  carry.  After  generating  the  multiplier  for  a specific 
problem  group,  the  multiplicand  is  generated.  This  (left)  factor  depends  completely  on  the  single-digit 
value(s)  of  its  according  factor  (multiplier). 

Table  2.  Probabilities  of  occurrence  of  digits  for  multipliers  of  problem  group  1. 


Digit 

Probability  of  appearance 

1 

5% 

2 

32% 

3 

32% 

4 

21% 

5-9 

10% 

The  algorithm  deals  with  random  numbers  and  probabilities  to  generate  appropriate  digit  values  for  the 
multiplicand  factor.  See  Table  3 for  illustration  where  the  beforehand-generated  possible  values  for  a 
multiplier  for  problem  group  1 are  shown  with  its  equivalent  domain  of  definition  for  the  multiplicand.  It  is 
mentioned  that  the  outermost  digit  to  the  left  of  a multiplicand  is  always  generated  randomly  without 
considering  the  multiplier’s  value  because  this  special  digit  can  also  include  a special  form  of  carry.  In  this 
case  the  carry  is  not  considered  for  the  number  of  successful  attempts. 
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Table  3.  Probabilities  of  occurrence  of  digits  for  multipliers  of  problem  group  1. 


Digit 

Multiplicand  (exclusive  of  left 
outermost  digit) 

1 

Random  number  out  of  [3,  9] 

2 

Random  number  out  of  [3,  4] 

3 

Random  number  out  of  [1,  3] 

4 

2 

5-9 

Random  number  out  of  [0, 1] 

Multipliers  from  expertise  category  II  and  III  are  not  generated  digit  by  digit.  They  were  drawn  at  random 
from  a table  of  all  remaining  possible  factors. Generating  factors  for  the  category  “multiplications  with  carry” 
is  easier  and  the  values  are  more  variant  but  the  generation  happens  in  a similar  way  as  explained  above. 

2.2  Evaluation  of  Multiplications 

In  table  4 all  positive  (OK)  states  for  the  different  sorts  of  calculation  processes  in  multidigit  multiplications 
are  shown.  In  the  same  way  table  5 shows  all  possible  error  states  which  the  program  is  able  to  identify. 


Table  4.  All  determinable  OK  states  and  their  possible  occurrences  in  the  program. 


Occurrence 

OK  state 

Description 

Examples  without  a carry 

OK 

OK  (failure-free) 

Examples  including 

OK 

OK  (failure-free) 

a carry 

OK_MENTAL 

OK_NOCARRY 

OK_ERR 

OK  (mental 
arithmetic) 

OK  (mental 
arithmetic  with 
succeeding  failure) 
OK  (including 
consequential  failure) 

Addition 

OK 

OK_END_ERR 

OK  (failure-free) 
Addition  OK 
(including 

consequential  failure) 

Overall  state 

OK 

NO_ERROR 

OK  (failure-free) 
No  failure 

Immediately  after  a user  responds  to  a multiplication,  the  algorithm  for  evaluation  of  all  detailed  user 
inputs  is  executed.  Beside  of  saving  absolutely  all  user  values  of  each  input  cell,  the  algorithm  takes  note  of 
specific  errors  and  the  so-called  positive  states  of  the  user  input.  The  main  objective  of  the  evaluation  process 
is  to  find  out  and  classify  typical  failures  done  by  each  learner.  The  evaluation  algorithm  for  multiplications 
including  carry  is  much  more  complex  than  the  one  for  simple  multiplications.  This  is  due  to  the  fact  that 
multiplications  with  carries  offer  different  input  values  for  different  strategies.  If  the  result  is  near  to  the 
correct  one,  a reconstruction  of  all  steps  which  has  been  well  done  is  not  always  possible.  Each  “OK”-  and 
“Error”  state  presented  in  the  tableS  is  totaled  in  the  profile  of  each  single  student.  These  states  are  the  basis 
for  the  statistical  presentation  and  hints.  Eor  teachers  the  hints  can  be  extremely  helpful  because  the  teachers 
have  no  opportunity  for  such  a close  look  on  a single  student  under  normal  circumstances. 
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Table  5.  All  determinable  errors  the  program  identifies. 


Occurrence 

Error  state 

Description 

Examples  without  a carry 

ERRJNPUT 

ERR_lxl 

ERR_REV_IN 

Input  failure 
Basics  failure 
Reverse  input  failure 

Examples  including 

ERRJNPUT 

Input  failure 

a carry 

ERR_lxl 

ERR_REVJN 

ERR_NOADD 

ERR_ADD 

ERR_UNKNOWN 

Basics  failure 
Reverse  input  failure 
Carry  error  (correct 
basics  - no  addition) 
Carry  error  (despite 
correct  answer) 
Unknown  failure 

Addition 

ERRJNPUT 

ERR_END_ADD 

ERR_END_OK 

Input  failure 
Addition  failure 
Addition  failure 
(despite  correct 
answer) 

Overall  state 

ERR_MISC 

Multiple  failure 

2.3  Adaptive  Learning  Algorithm 

When  the  program  starts  it  is  a real  challenge  to  estimate  the  competency  level  of  the  actual  user  in  our 
hierarchy  of  16  categories  in  order  to  generate  an  adequate  multiplication  task  that  suits  the  user’s  current 
expertise. 

For  each  user  the  algorithm  sets  six  so-called  expertise  attributes  in  the  database.  These  attributes  reflect 
the  current  user’s  knowledge  in  multidigit  multiplication.  The  six  values  are  divided  into  two  groups  of  three 
categories  - one  table  for  simple  multiplications  without  carry  and  the  other  table  for  multiplications  with  a 
carry.  The  three  categories  reflect  the  three  classes  of  expertise  in  table  1,  respectively. 

At  the  beginning  of  an  assessment  the  current  attribute  values  of  a user  are  loaded  from  the  database.  If  it 
is  the  first  time  the  user  is  going  to  practice,  all  values  are  initialized  to  zero.  The  learning  algorithm  now 
chooses  randomly  a multiplication  of  the  simplest  expertise  category  (out  of  problem  group  1 to  3),  as  well 
randomly  without  or  including  a carry.  The  multiplication  is  then  displayed  to  the  user.  After  solving  the 
multiplication,  the  evaluation  algorithm  computes  the  state  of  the  user  input  (the  result)  and  displays  the  user 
if  his  or  her  answer  was  correct  or  incorrect.  In  the  background  the  expertise  attribute  matching  the  expertise 
category  of  the  generated  multiplication  is  increased  with  1,  if  the  user’s  answer  was  correct,  or  decreased 
with  1,  if  the  answer  was  incorrect.  If  the  attribute  value  was  zero,  no  decrease  happens.  The  same  happens 
for  a value  equal  to  three  - no  increase  takes  place.  Next,  another  example  is  generated  from  the  lowest 
expertise  category.  If  an  expertise  attribute’s  value  is  equal  to  three,  this  means  that  the  user  has  practiced 
three  consecutive  solutions  and  it  is  supposed,  he  or  she  is  now  able  to  solve  multiplications  from  the 
reflected  expertise  category.  In  the  following  process,  no  more  examples  from  this  category  are  generated.  If 
all  six  attributes  are  set  to  value  three  for  a specific  user,  then  this  user  has  successfully  solved  the  training 
mode.  These  users  have  the  expertise  to  solve  all  multidigit  multiplications  in  this  domain.  At  any  time 
during  the  training  a so-called  training  example  is  generated  and  displayed  to  the  user  (10%  of  occurrences). 
Such  a training  example  is  simply  a multiplication  chosen  from  the  already  solved  expertise  categories.  This 
serves  as  a refreshment  of  knowledge  and  leads  to  strengthening  the  training  purpose.  If  no  multiplications 
were  solved  correctly  and  a training  example  is  chosen  randomly,  then  an  alert  for  the  teacher  is  generated. 


3.  TECHNICAL  DETAILS 

The  web-based  bilingual  application  is  programmed  in  PHP  5 using  the  Zend  Framework.  For  layout  and 
design  purposes  Cascading  Style  Sheets  (CSS)  and  jQuery  are  used.  MySQL  was  chosen  as  a relational 
database  management  system.  All  mentioned  technologies  are  Open  Source  and  can  be  freely  adapted  and 
distributed. 
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4.  USER  INTERFACE 

First  of  all,  there  is  a start  page  where  users  are  able  to  log  in  with  a specific  username  and  password.  To  be 
able  to  log  in  and  to  play  with  the  multiplication  training  program,  users  have  at  first  to  register  for  free  on  a 
separate  webpage  (http://mathe.tugraz.at).  After  the  login  process  students  enter  a welcome-page  (“Are  you 
ready?”)  as  a preparation  for  the  training  mode.  Logged-in  teachers  will  see  an  additional  “Statistics”  button 
at  the  top  of  the  page.  When  clicking  on  the  green  button  (“Let’s  go”),  the  training  mode  starts  and  according 
to  the  described  algorithm  an  appropriate  example  is  presented. 

Figure  1 shows  a correct  result  of  the  overall  calculation  (user  inputs)  of  a multiplication  of  expertise 
category  II  without  add  carry.  All  input  cells  were  checked  and  marked  correctly  in  the  evaluation  process. 
The  evaluation  algorithm  therefore  computes  a statistic  on  positive  result  states,  which  have  been  saved  in 
correspondent  database  tables.  Additionally,  the  learning  algorithm  increases  the  user’s  property  for  expertise 
category  II  (compare  the  two-digit  multiplier)  by  adding  I to  his/her  state. 


Figure  1 . Correctly  solved  multidigit  multiplications. 

Figure  2 demonstrates  a wrong  answer  of  a filled-in  multiplication  calculation  of  the  most  difficult 
problem  group  (8)  including  carry.  Wrong  input  cells  are  marked  with  a red  background  to  advise  users 
exactly  where  failures  occurred.  The  evaluation  algorithm  has  computed  result  states  for  each  input  cell,  as 
well  as  an  overall  result  state.  The  second  row’s  result-states  include,  for  example,  correct  calculations  (OK), 
a correct  mental  calculation  (OK_MENTAL)  as  well  as  the  state  OK_NOCARRY  and  an  unknown  error 
(ERR_UNKNOWN).  The  third  result  row’s  outcome  states  include:  OK,  ERR_NOADD,  ERR_UNKNOWN, 
OK_ERR  (see  tables  above  for  an  explanation  of  specific  result  states). 


Figure  2.  Wrongly  answered  multidigit  multiplications. 
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Figure  3 points  out  the  statistics  of  all  grades  and  students  belonging  to  a teacher/class.  Each  student  is 
listed  with  its  name,  total  amount  of  solved  examples,  total  amount  of  correct  solutions,  total  amount  of 
incorrect  solutions,  as  well  as  a computed  success  rate  and  error  burst.  A positive  success  rate  (>  70%)  is 
marked  with  a green  background  whereas  a negative  success  rate  is  marked  with  red  background.  A link 
leads  to  detailed  table  for  a selected  student.  This  figure  shows  the  typically  error  bursts; 

• Basic  errors  (less  competence  in  handling  the  multiplication  tables) 

• Problems  with  the  carry 

• Multiple  Errors 


GRADE  saaralt 


User  name 

Overall 

Correct 

Incorrect 

Success 

Error  burst 

Details 

answers 

answers 

answers 

rate 

BastiOS 

Never 

21 

14 

7 

66  67% 

Mutople  errors  > HERE  < 

Anton  Cary 

11 

7 

4 

63  64% 

Carry  addition  error  (correct  basics  - no  > HERE  < 
addition) 

nolxl 

minir>g 

22 

9 

13 

40  91  % 

Basics  error  > HERE  < 

Figure  3.  List  of  learners  and  compressed  statistics  (Error  burst). 


3868’*48 

Basic 

multiplication 

Correct  carry  to 
add 

Correa 

Soluoon 

User  carry  input  to 
add 

User 

answer 

New  carry 

Result  status 

8»4 

OFF 

32 

OFF 

32 

3 

OK 

6*4 

3 

27 

3 

27 

2 

OK 

8*4 

2 

34 

2 

34 

3 

OK 

3*4 

3 

15 

3 

15 

OFF 

OK 

Figure  4.  Detailed  analysis  of  one  task. 

Figure  4 shows  the  top  of  the  most  detailed  listing.  (This  is  only  optionally  shown!)  It  includes  each 
single  input  (cell)  of  a single  multiplication  problem.  It  shows  exactly  the  evaluation  of  the  generated 
problem  of  figure  2,  including  the  computed  positive  and  erroneous  states  as  explained  before. 


5.  CONCLUSION 

The  ubiquity  of  laptops,  netbooks,  tablet  computers,  and  smartphones  among  young  people  led  to  a complete 
new  way  of  dealing  with  learning  materials  as  well  as  watching  learning  processes.  Research  in  the  field  of 
Learning  Analytics  seems  to  be  a big  task  in  the  future.  For  example,  many  teachers  express  their  fears  about 
being  replaced  or  to  be  less  or  not  necessary  anymore.  Scandura  [11]  spread  in  his  article  exactly  this  issue 
and  answered  immediately;  "What  TutorIT  can  do  better  than  a human  and  Why;  Now  and  in  the  Future". 
Nevertheless,  our  research  team  is  not  sharing  this  opinion.  The  role  of  tomorrow's  teachers  is  much  more 
like  to  facilitators  and  coaches  who  assist  learners  according  to  their  individual  needs.  Many  teachers  are 
offering  other  hints  and  additional  material.  Our  application  should  have  the  role  of  such  an  additional 
content.  We  don’t  dictate  the  students  the  individual  solution  steps.  If  anyone  prefers  only  mentally 
computing,  the  right  solution  is  accepted.  If  someone  does  not  use  the  prepared  input  fields  for  the  carries  , 
this  will  be  reported. 

Furthermore  our  application  gathers  more  details  about  learning  multidigit  multiplication.  Resulting  from 
many  personal  discussions  with  teachers  about  common  failures  our  team  got  the  impression  that  teachers 
almost  exclusively  see  only  one  cause  of  problems;  multiplication  facts.  Due  to  the  fact  that  teachers  usually 
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have  no  time  to  take  a closer  look  or  to  figure  out  systematically  what  might  have  caused  each  single  failure 
this  application  should  close  this  gap.  The  program  tests  not  only  for  competence  in  multiplication  facts. 
Additionally  it  takes  also  a look  on  other  fundamental  skills.  Finally,  if  teachers  use  the  advice  of  the 
program,  they  can  try  to  help  the  students  in  targeted  manner.  In  some  cases,  the  students  will  probably  have 
other  physiological  and  psychological  problems  which  were  presented  in  detail  in  in  Dowker  [3]. 

For  the  future  of  the  program  also  the  issue  “processing  time”  is  a challenge.  This  is  corresponding  to  the 
goal  of  Hasselbring  [7]:  an  “automatic  level  of  proficiency”.  This  is  a category  that  would  be  easy  to 
implement  in  a diagnostic  system.  In  a close  cooperation  to  further  interested  institutions  this  issue  as  well  as 
a huge  field  study  will  be  carried  out  in  future.  It  can  be  concluded  that  Learning  Analytics  in  combination 
with  the  possibilities  of  innovative  technologies  and  innovative  teaching  and  learning  strategies  is  a powerful 
field  to  enhance  learning  processes  of  each  individual  learner. 

The  apps  and  the  user  account  for  this  and  other  little  helpers  are  available  for  free  on 
http://mathe.tugraz.at . Choose  “Registrieren”  and  then  “English”.  With  a single  e-mail  address  you  can  get 
an  account  to  test  the  functionality.  To  get  accounts  for  schools  and  classes  please  have  contact  with 
martin.ebner@tugraz.at. 
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ABSTRACT 

Microblogs  have  been  used  in  the  educational  context.  However,  differently  from  following  friends’  status  messages,  in 
the  educational  scenario  it  is  important  to  follow  the  discussions,  to  understand  the  flow  of  messages.  YouFlow 
microblog  was  developed  for  that  purpose.  It  is  a microblog  that  provides  the  main  structures  of  discourse  that  are 
available  on  communication  systems  as  well  as  messages'  categorization  according  to  a lesson  plan.  An  exploratory  case 
study  allowed  analyzing  the  use  of  the  structures  of  discourse  on  the  microblog  in  an  educational  context.  Then,  an 
explanatory  case  study  showed  an  increase  in  the  participation  on  discussions  of  topics  according  to  a lesson  plan. 

KEYWORDS 

microblogs;  discussions;  collaborative  learning;  lesson  plan;  structure  of  discourse. 


1.  INTRODUCTION 

As  microblogs  become  more  used,  some  differences  are  explored  on  specific  domains,  such  as  microblogs 
for  a specific  geographic  area:  i)  Frazr  (http://www.frazr.com),  which  is  focused  on  European  public  that 
speaks  German,  Spanish  or  French,  and  ii)  Sina  (http://www.sina.com.cn),  focused  on  the  Chinese  public;  or 
a specific  age  group  or  sex  (Sina  - http://www.sina.com.cn),  common  objective,  learning  (Edmodo  - 
http://www.edmodo.com,  and  Cirip  - http://www.cirip.ro),  collectivity  and  other  characteristics. 

Similarly,  some  online  social  network  environments  such  as  Facebook  (http://www.facebook.com)  and 
Myspace  (http://www.myspace.com)  provide  microblog  functionalities  for  the  participants.  An  analysis  of 
the  100  main  learning  systems  in  2009  showed  the  importance  of  microblogs  systems  in  the  educational 
context  as  Twitter  (http://www.twitter.com),  Edmodo  and  Plurk  (http://www.plurk.com)  were  listed 
(Holotescu  and  Grosseck,  2010  ).  In  2011  (http://c41pt.co.uk/top-100-tools-for-learning-2011/),  Twitter  is 
listed  as  the  main  system  (among  the  top  100)  to  support  learning. 

Microblogs  are  systems  with  a high  publishing  frequency,  which  makes  the  conversation  interesting  for 
learning.  Although  they  are  considered  asynchronous  conversation  tools,  usually  there  is  an  expectation  that 
the  information  is  consumed  in  real  time  because  there  is  a huge  amount  of  users  online. 

However,  this  high  amount  of  publications,  in  addition  to  the  way  the  messages  are  organized,  may  lead 
to  a misunderstanding  of  the  discussion  (Lim,  2008).  It  promotes  a search  for  other  microblogs  that  can 
organize  the  content  better  (Academic  Aesthetic,  2008). 

As  in  a microblog  the  publishing  frequency  is  high  (140  million  of  posts  per  day  in  Twitter  - 2011 
KISSmetrics  - http://blog.kissmetrics.com/twitter-statistics/),  it  is  necessary  a filtering  mechanism  for  the 
user  to  get  the  relevant  content  and  follow  the  discussion.  When  applied  to  the  educational  context,  the 
relationships  among  the  messages  (posts,  comments,  replies  etc.)  is  really  important  because  the  user  needs  to 
understand  not  only  the  message  content,  but  also  its  relationship  with  the  other  messages  in  order  to  follow 
the  discussion  and  participate  of  the  collaborative  dynamics  based  on  discussions  on  microblogs.  In  the 
future,  this  kind  of  dynamics  could  be  supported  by  models  such  as  those  proposed  by  Atif  (2012). 

We  explored  the  discourse's  structures  as  well  as  the  categorization  of  messages  in  microblogs  for 
teaching  and  learning.  Section  2 presents  structures  of  discourse  in  microblogs  according  to  the  exploratory 
case  study.  Section  3 explores  how  categorization,  filtering  and  search  can  be  used  in  the  educational  context 
in  microblogs.  Section  4 presents  the  data  analysis  of  our  research  according  to  the  use  of  these 
functionalities  on  YouFlow  during  the  second  case  study.  Section  5 provides  a deeper  analysis  on  the 
participation  of  the  students  on  the  second  case  study.  Finally,  Section  6 presents  some  final  remarks. 
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2.  STRUCTURES  OF  DISCOURSE  ON  MICROBLOGS 

We  supposed  the  structure  of  discourse  may  influence  on  how  the  user  can  follow  the  messages  in 
microblogs.  Then,  this  section  describes  the  structures  of  discourse  that  are  available  on  collaborative 
(communication)  systems.  The  structuring  of  discourse  represents  the  relationship  possibilities  among  the 
messages.  Among  the  main  structures  are  (Gerosa  et  al,  2005):  list,  tree  and  graph. 

In  the  list  structure  (or  linear)  each  message  is  chained  sequentially.  It  may  be  ordered  by  the  author's 
name,  message's  title  or  posting  date.  The  visualization  of  the  messages  follows  this  order  and  therefore  the 
possible  relationships  among  the  messages  are  not  explicit  (although  they  may  happen,  for  instance  a 
message  may  be  an  answer  to  another  one). 

In  the  tree  structure  (or  hierarchical),  each  message  (that  comes  after  the  first  one)  must  be  associated  to 
another  one  that  is  called  its  parent.  A parent  message  may  be  associated  to  more  than  one  child.  It  allows  a 
divergence  on  the  communication. 

In  the  graph  structure  (or  network),  a message  may  be  associated  to  more  than  one  message  at  the  same 
time  (independently  of  the  hierarchical  position).  Therefore  it  is  possible  to  notice  when  a conversation  is 
getting  a consensus.  This  approach  is  similar  to  the  one  presented  at  (Camusi  and  Giovannella,  2010). 

The  star-graph  structure  is  a specific  type  of  the  tree  structure  where  there  is  a central  node  (root),  which 
is  the  parent  of  all  the  other  messages.  The  root  message  is  the  one  that  starts  the  discussion.  All  the  other 
messages  are  called  leaves.  This  structure  allows  a focused  discussion  (on  the  starting  message)  and  it  is 
common  on  blogs  (Marques  et  al,  2013). 

YouFlow  microblog  was  developed  to  support  an  analysis  on  the  different  structures  of  messages. 
Therefore  one  can  choose  the  structure  and  use  it  accordingly.  Table  1 compares  YouFlow  and  other 
microblogs  used  in  the  educational  context  according  the  structuring  of  messages.  Most  of  them  are  based  on 
the  linear  and  star-graph  structures. 


Table  1.  Comparing  the  structuring  of  discourse  on  the  main  microblogs 


Microblog 

List 

Star-Graph 

Tree 

Graph 

Cirip 

X 

Edmodo 

X 

X 

Twiducate 

X 

Twitter 

X 

X 

Plurk 

X 

X 

Buzz 

X 

X 

YouFlow 

X 

X 

X 

X 

Figure  1 shows  the  Tree  structure  in  YouFlow,  while  Figure  2 show  the  Graph  structure. 
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^ C leber  | Posted  3 months  ago 

Is  the  relational  model  a logical  data  model? 

alberto  | Posted  2 mnutes  ago 

Rails  is  an  x Q^k  of  frameworks), 

cornposed  alberto  says:  Rails  is  an ’meta-frameworlc"  d.  Action  Pack,  Action 

Mailer,  Acti  ^ , , 

(that  IS,  It  IS  a framework  of  frameworks  , 

9 

A Debora  | Posted  3 months  ago 

Active  Record,  Action  Pack,  Action  Mailer, 
Active  Support  and  Action  Web  Services 

In  my  opinion,  they  are  different.  The  relational  model 
represents  the  base  of  database,  and  the  logical  data  model, 
the  business,  not  only  the  database.  But  I'm  not  sure. 

lucas  1 Posted  3 mnutes  ago 
What  is  "Rails"? 

m 

^ flavk>  1 Posted  less  than  a mnute  ago 

* Hi  Lucas,  have  a look  in  the  definition  that  Alberto  wrote  about  the 

A Wagner  (Posted  3 months  ago 

According  to  my  understanding,  the  relational  model  would 
be  the  logical  data  model.  This  is  what  the  book  says. 

9 

Figure  2.  Graph  structure  in  YouFlow. 

Q C leber  | Posted  3 months  ago 

Yes,  one  of  the  steps  of  the  logical  data  model  is  to 
build  the  relational  model. 

•_  . 

A Wando  | Posted  3 rr>onths  ago 

According  to  my  readings,  building  the  relational  model 
allows  the  designer  to  create  a logical  model  that  is 
consistent  with  the  information  to  be  stored. 

9 

Mateus  | Posted  3 months  ago 

i Thus,  facilitating  building  and  understanding  the 

databases  that  are  used  in  the  enterprise,  because  it 
considers  the  data  that  are  used. 

■) 

Figure  1.  Tree  structure  in  YouFlow. 


A case  study  using  YouFlow  microblog  in  an  educational  context  was  performed  during  the  first  semester 
of  2011  in  the  course  of  Fundamentals  of  Information  Systems.  This  course  is  part  of  the  undergraduate 
curriculum  of  Information  Systems  of  a university. 

From  the  data  analysis  it  was  possible  to  notice  the  importance  of  using  microblogs  in  the  educational 
context.  When  inquired  about  the  microblog  influence  in  learning  (closed  question),  82%  of  the  students 
answered  it  favors  learning  and  18%  answered  it  does  not  influence.  In  the  testimonies  (open  questions)  it 
was  also  possible  to  notice  the  microblogs'  influence  on  learning.  Maria  stated  "the  use  of  microblogs  helps 
on  improving  the  interaction  among  the  students,  directly  contributing  to  increase  everyone's  participation.  It 
makes  the  activity  delightful.".  Lucas  said  he  "liked  it  because  it  promotes  a good  interaction  and  improves 
searching  online  content".  John  told  he  "thinks  it  is  good  for  exchanging  information  about  the  course". 
Finally,  Cleber  stated  "it  is  interesting  because  it  makes  the  students  leaves  the  inertia.". 

About  the  use  of  the  microblog,  the  participants  highlighted  its  simplicity,  dynamics,  practice  and 
intuitive  use  as  the  characteristics  they  liked  most.  They  answered  the  open  questions  with  good 
observations.  John;  "YouFlow  is  quite  similar  to  Twitter.";  Lucia:  "I  liked  most  the  messages  about  the 
subjects  and  the  possibility  of  quoting  comments  of  other  friends.";  Maria:  "I  liked  the  fact  of  everybody 
participating  more  since  the  interaction  through  the  computer  decreases  the  inhibition  of  the  students.";  and 
Cleber:  "it  is  a different  (interesting)  way  of  studying.". 

Although  the  participants  were  used  to  the  environment  based  on  microblog,  some  had  some  difficulties. 
From  the  participants  that  answered  the  questionnaire,  18%  of  them  thought  it  was  very  easy  to  use,  55% 
almost  did  not  have  difficulties  and  27%  had  some  difficulties.  Through  the  log  analysis  it  was  possible  to 
notice  that  the  quote  functionality,  which  was  available  in  the  structuring  of  discourse  based  on  graph,  was 
not  used.  John  reported  he  "had  some  problems  with  the  user  interface  that  confused  in  some  moments." 
Carla  said  she  "had  difficulty  on  using  graph".  The  other  approaches,  linear  and  tree,  were  commonly  used. 
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Although  the  graph  structure  provides  more  possibilities  for  relationships  among  the  messages  (quotes),  it 
was  considered  a complex  alternative.  Perhaps  it  would  be  necessary  to  consider  a previous  training  with  the 
participants  in  order  to  have  this  functionality  used. 

The  linear  structuring  of  discourse  promoted  the  communication  through  publishing  content  and 
definitions.  However,  when  the  focus  is  on  discussions,  the  relationships  among  the  messages  are  important. 
This  was  the  case  for  one  participant  when  making  explicit  the  relationship  through  the  use  of  " @ " sign  for 
answering  a message  from  other  participant  (this  is  a really  common  feature  on  Twitter).  The  tree  structure 
was  the  alternative  that  best  adapted  to  the  dynamics  because  it  allowed  the  relationship  among  the  messages 
and  promoted  the  interaction  among  the  participants. 


3.  CATEGORIZATION,  FILTERING  AND  SEARCH  IN  MICROBLOGS 
IN  THE  EDUCATIONAL  CONTEXT 

Collaboration  requires  communication,  coordination  and  cooperation  (Gerosa  et  al,  2005).  Individuals  must 
collaborate  in  order  to  work  or  learning  in  group.  They  must  share  ideas  (communicate),  work  in  harmony 
with  other  participants  of  the  group  (coordinate)  and  perform  tasks  well  (cooperate)  (Fuks  and  Assis,  2001). 

Although  the  discussion  of  subjects  is  important  to  communication  systems,  when  it  is  not  correctly 
managed  it  may  result  on  information  overload.  It  can  bring  out  the  difficult  of  assimilating  content. 

In  order  to  reduce  the  content  provided  to  the  user,  predefined  criteria  can  be  used  to  categorize  content. 
We  chose  the  classification  based  on  topics  of  a class  plan  (programmatic  content)  because  the  domain 
specialist  (the  teacher)  defines  the  topics  to  be  discussed.  It  promotes  thinking  on  the  students’  messages  and 
provides  a classification  that  is  better  or  equal  to  the  free  classification.  The  classification  based  on  the  topics 
of  the  programmatic  content  of  a class  plan  also  provides  different  mechanisms  for  retrieving  information  as 
well  as  it  has  a good  potential  to  support  teaching  and  learning. 

The  classification  of  the  messages  can  support  search  and  retrieving  information  through  filters  that  helps 
understanding  what  is  under  discussion  or  through  textual  searches. 

Therefore,  the  message  categorization  in  the  microblog  YouFlow  is  based  on  the  topics  of  a class  plan. 
When  registering  a new  topic,  information  about  the  topic  (such  as  the  description  of  the  topic)  can  also  be 
added.  Then,  textual  searches  can  use  these  pieces  of  information  to  improve  and  expand  their  results. 

For  instance,  a class  plan  could  be  organized  in  topics  such  as  Technology-Enhanced  Learning, 
Web-Based  Education,  and  Microblogs  for  Learning.  Then,  these  topics  should  be  registered  in  YouElow  and 
when  posting  a message,  the  user  could  choose  one  of  these  topics  to  categorize  its  message.  Therefore,  it 
would  be  possible  to  filter  the  messages  related  to  each  specific  topic. 

The  textual  search  is  based  on  keywords,  which  allows  retrieving  the  categories  that  has  in  their 
description  the  searched  term.  Then,  users  may  have  access  to  microblog  messages  that  do  not  explicitly 
mention  the  searched  term,  but  that  were  classified  in  a category  that  has  this  term  in  its  description.  The 
keyword  search  may  be  executed  through  a free  textual  search  or  through  categories  that  were  predefined 
according  to  the  class  plan. 

The  search  by  related  messages  is  executed  in  three  steps:  i)  retrieving  information  from  tracks  that  have 
the  searched  term,  ii)  recognition  of  each  node  of  the  track  that  corresponds  to  the  search,  and  iii)  exhibition 
of  the  sub-track  that  is  built  from  the  previous  step. 

Therefore,  YouElow  microblog  has  categorization,  filtering  and  search. 


4.  SECOND  CASE  STUDY 

A second  case  study  was  performed  aiming  at  analyzing  if  in  a microblog  with  tree  structuring  of  messages, 
in  addition  to  the  classification  of  its  messages  in  topics  of  a class  plan,  would  provide  a better  following  of 
the  discussion  as  well  as  better  participation  of  the  students. 

The  case  study  was  performed  in  the  second  semester  of  201 1.  It  started  in  a classroom  of  Eundamentals 
of  Information  Systems,  of  the  undergraduate  program  of  Information  Systems  in  a university.  The 
characteristics  of  the  students  and  the  class  are  the  same  of  the  first  case  study.  In  this  class,  YouElow 
microblog  was  introduced  to  the  students  as  well  as  the  instructions  for  the  class  activity. 
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It  was  possible  to  notice  from  the  application  log  that  the  amount  of  messages  by  day  was  bigger  when 
considering  categorization  of  the  messages  than  the  approach  with  no  categorization  of  messages.  Fig.  3 
shows  the  diagram  of  discussions  per  day  and  the  amount  of  messages  in  each  approach. 


With  * without  categorization 


With  categorization 

Without  categorization 


Figure  3.  Messages  per  day. 

In  addition  to  the  amount  of  messages  with  categorization  being  bigger  than  the  amount  without 
categorization  of  the  messages,  the  students  confirmed  the  preference  to  the  categorization  approach  through 
the  online  questionnaire  with  open  and  closed  questions.  When  inquired  about  the  approach  that  benefits  the 
education,  80%  of  the  students  selected  the  categorization  while  20%  chose  the  approach  without 
categorization.  When  inquired  about  the  approach  that  improves  the  discussion,  the  result  was  similar  to  the 
education  question;  70%  selected  the  categorization  approach  while  30%  chose  the  approach  with  no 
categorization. 

The  categorization  is  available  at  two  stages:  i)  publishing  and  ii)  reading.  When  inquired  about 
publishing  messages,  30%  of  the  users  preferred  to  publish  without  classification  while  70%  preferred  to 
publish  the  message  classifying  it.  When  inquired  about  reading,  90%  of  the  users  preferred  to  use  the  filter 
to  understand  the  discussion,  while  10%  preferred  not  to  use  the  filter  and  see  everything.  John  argues  about 
the  importance  of  using  the  filter  especially  when  reading:  "It  is  easier  than  in  other  blogs.  I always  got  lost 
on  the  blogs  because  we  are  in  a discussion  and  someone  sends  a message  that  is  not  related  to  the 
discussion.  Then,  another  person  answers  that  and  sometimes  the  previous  subject  gets  lost.  The  filter 
organizes  the  messages  and  solves  this  problem.". 

The  open  questions  of  the  questionnaire  allowed  collecting  data  about  the  educational  dynamics.  When 
inquired  about  the  experience  of  using  YouFlow  microblog  on  the  course,  the  results  show  the  use  of 
microblog  improved  learning.  Some  answers  were:  "It  showed  a usage  that  I hadn't  thought  about  to  discuss 
the  subjects  in  a faster  way.",  "It  was  an  interesting  experience  because  I already  use  microblogs,  but  in  a 
different  goal  from  the  activity  in  YouFlow.",  "It  was  an  excellent  experience  of  sharing  information  and 
learning  that  should  be  used  since  the  beginning  of  the  courses  because  it  really  helps  the  students.",  "It  is 
interesting  because  the  course  has  lots  of  subjects  and  plenty  of  content  material.  The  subjects  lead  to 
discussions  and  the  microblog  helped  the  discussion  outside  the  classroom,  online.". 

The  project  of  the  case  study  consisted  of  using  YouFlow  in  three  stages:  there  were  two  different 
subjects  (stages  one  and  two)  to  be  discussed  and  the  third  stage  used  a subject  that  encompassed  the 
previous  two.  Therefore,  in  the  third  stage  the  students  could  use  the  search  functionality  available  on  the 
microblog  to  access  the  previous  content.  In  the  stage  one,  the  groups  of  students  were  organized  on  those 
who  would  discuss  the  subject  using  the  categorization  of  messages  and  those  who  would  discuss  the  subject 
without  categorizing  the  messages.  Then,  in  the  stage  two,  the  group  exchanged  the  approach  (i.e.,  the  one 
who  used  categorization  of  the  messages  in  stage  1 could  not  categorize  the  messages  in  stage  2 and  vice- 
versa). 

The  students  were  at  the  beginning  of  the  course  and  therefore  they  were  in  at  the  same  level.  They  were 
motivated  to  participate  on  the  discussion  of  the  group  and  the  results  presented  at  (Stepanyan  et  al,  2010)  for 
student  interaction  at  Twitter  does  not  directly  apply  to  this  study. 

Through  the  analysis  of  the  application  log,  it  was  possible  to  notice  that  the  search  functionality  was  not 
used  as  much  as  expected.  There  were  only  a few  searches.  However,  when  inquired  about  the  search  results. 
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50%  of  the  users  preferred  the  categorization  of  the  messages  (topics),  50%  preferred  related  messages  and 
0%  preferred  the  loose  messages  when  the  focus  was  on  the  discussion.  Considering  the  learning  aspect,  80% 
of  the  users  preferred  the  categorization  (topics),  20%  preferred  the  related  messages  and  0%  preferred  the 
loose  messages.  Although  preliminaries,  the  search  results  show  a possible  evidence  that  the  search  results 
grouped  by  categories  and  related  messages  are  more  relevant  (or  better)  to  discussion  and  education 
purposes. 


5.  ANALYSIS  ON  THE  IMPROVEMENT  OF  THE  DISCUSSIONS 

Be  P = {A,  B,  C,  D,  E,  F}  the  graph  (tree)  showed  in  Fig.  4.  The  tree's  depth  is  the  depth  of  the  leave  of 
highest  hierarchy,  i.e.,  leaves  E and  F.  For  the  tree  P,  the  depth  is  equal  to  2. 

For  the  data  triangulation,  two  groups  were  analyzed:  with  and  without  categorization.  Considering  the 
tree  depth  concept,  the  depth  of  a discussion  with  and  without  categorization  was  calculated  through  the  sum 
of  each  tree  of  the  forest,  divided  by  the  amount  of  trees.  Fig.  5 shows  the  formula  for  calculating  the  average 
depth  according  to  each  approach  (with  and  without  categorization),  where  Pm  is  the  average  depth  of  each 
approach,  and  Pi  is  the  depth  of  each  discussion  on  each  approach,  varying  from  i=l  to  n,  where  n is  the 
maximum  amount  of  tracks  of  the  approach. 


Figure  4.  Depth  of  a tree. 


We  analyzed  the  data  from  the  application  log  and  applied  the  formula  for  calculating  the  average  depth 
by  approach.  The  discussions  based  on  the  tree  structure  with  categorization  of  messages  had  a higher 
average  depth  than  the  one  without  categorization.  Table  2 shows  the  results  of  average  depth,  where  in  the 
categorization  we  found  2,12  messages  while  without  categorization  we  found  1,43  messages.  The 
categorization  approach  had  53  messages  in  a forest  of  17  trees.  The  approach  without  categorization  had  17 
messages  in  a forest  of  7 trees. 


Table  2.  Average  depth  by  approach 


Approach 

Average  depth 

Amount  of  messages 

Amount  of  trees 

With  categorization 

2,12 

53 

17 

Without  categorization 

1,43 

17 

7 

Analyzing  the  17  discussions  (trees)  from  the  categorization  approach,  the  maximum  depth  was  5 (first 
and  fifth  discussions),  while  the  minimum  depth  was  0 (second  discussion).  The  depth  zero  means  a message 
was  published,  but  had  no  answer. 

Analyzing  the  7 discussions  (trees)  from  the  approach  without  categorization,  the  maximum  depth  was  2 
(third,  fifth  and  seventh  discussions)  while  the  minimum  was  1 (all  the  other  discussions). 

Fig.  6 and  Fig.  7 show  the  probability  by  depth  for  each  approach  (respectively,  with  categorization  and 
without  categorization). 
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with  categorization 


0 1 2 3 4 5 

Depth 

Figure  6.  Probability  of  occurences  of  discussions  by 
depth  (with  categorization). 


without  categorization 


Depth 


Figure  7.  Probability  of  occurences  of  discussions  by 
depth  (without  categorization). 


In  this  research  we  considered  the  messages  that  are  related  to  the  root  node  in  each  discussion. 
Therefore,  the  amount  of  messages  for  each  track  (or  discussion)  per  approach  (with  categorization  and 
without  categorization)  was  calculated.  Table  3 shows  the  average  amount  of  messages  per  track.  When  the 
categorization  of  messages  was  available,  the  amount  of  messages  was  higher  than  when  the  discussions  did 
not  consider  the  categorization  of  messages. 


Table  3.  Average  of  messages  by  approach 


Approach 

Average  of  messages 

With  categorization 

1,9 

Without  categorization 

1,4 

6.  CONCLUSION 

In  this  work  we  explored  the  use  of  different  structures  of  discourse  as  well  as  the  categorization  of  messages 
in  microblogs  when  considering  the  educational  context.  Although  other  works  discuss  the  influence  of 
microblogs  in  the  academic  environment,  e.g.,  (Ebner,  2013)  and  present  the  importance  of  discourse 
structuring  in  microblogs,  e.g.,  (Miihlpfordt  and  Wessner,  2005),  we  developed  a microblog  that  allowed  not 
only  structuring  the  discourse  according  to  different  approaches,  but  also  characterizing  the  discussions 
according  to  a lesson  plan.  The  objective  was  to  provide  a better  understanding  of  the  discussions,  allowing 
the  participants  to  follow  the  messages  and  their  relationships  in  collaborative  dynamics  based  on 
discussions.  The  results  were:  i)  the  categorization  of  messages  according  to  the  class  plan  allowed  deeper 
discussions;  ii)  the  amount  of  messages  and  preference  of  the  students  for  the  categorization  approach  are 
directly  related;  iii)  the  filtering  of  messages  (categorization  on  the  reading  time)  was  used  as  a facilitator  of 
the  discussion  understanding;  iv)  the  students  prefer  search  results  organized  by  the  categories  and  related 
messages;  and  v)  the  students  were  able  to  follow  the  discussions  supported  by  the  tree  structure  of  discourse, 
categorization  of  messages  according  to  the  class  plan  and  searches  or  filtering  of  messages. 

Some  approaches  for  messages'  visualization  were  considered  in  this  work,  such  as  discourse  structuring 
(linear,  tree,  graph  and  star-graph),  categorization  of  messages  and  search.  However,  other  approaches  could 
be  explored,  such  as  i)  recommendation  of  messages  in  microblogs;  and  ii)  groupings  of  messages  according 
to  date,  priority  and  subject.  These  approaches  are  alternatives  for  reducing  the  amount  of  information  to  be 
explored  and  therefore  they  consist  of  possibilities  for  showing  the  messages  that  are  more  relevant  to  the 
user,  which  in  an  educational  context  can  provide  better  understanding  of  the  content  and  discussions. 

Although  the  focus  of  our  work  is  on  helping  the  participants  of  the  conversations  to  follow  the 
discussions,  we  could  also  provide  easier  feedback  to  the  teacher  in  an  approach  similar  to  the  MIRA  (Akbari 
et  al,  2010),  but  based  on  message  structuring  and  filtering. 

The  search  presented  by  YouFlow  has  a good  potential  to  be  explored  in  microblogs  used  for  learning, 
especially  considering  the  results  by  topics  of  a class  plan  and  related  messages.  If  the  domain  semantics  is 
also  considered,  then  we  believe  that  even  better  results  could  be  provided.  It  would  be  possible  to  provide 
search  results  based  on  synonyms,  meanings  or  related  subjects  or  even  other  categorizations  schemes  (e.g., 
user-based  categorization).  In  addition,  automatic  classification  procedures  such  as  an  adaptation  of  the 
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approach  used  for  classifying  learning  objects  presented  in  (Mendoza  and  Becerra,  2010),  (Lama  et  al,  2011) 
or  (Lau  and  Lee,  2012)  could  be  considered.  We  are  also  studying  if  YouFlow  can  be  considered  a new 
communication  mechanism:  the  micro-forum. 


ACKNOWLEDGEMENT 

This  work  was  partially  supported  by  FAPERJ  (grants  E-26/10 1.509/20 10  - BBP/Bursary  Representation  and 
contextualized  retrieval  of  learning  content,  and  E-26/170028/2008  INCc&T  Program  - Project:  Brazilian 
Institute  of  Research  on  Web  Science)  and  CNPQ  (project:  557.128/2009-9,  INCT  on  Web  Science). 


REFERENCES 

Academic  Aesthetic,  2008.  Academic  Aesthetic  165:  Twitter  vs  Plurk.  Available  at: 

http://academicaesthetic.eom/2008/08/06/academic-aesthetic-165-twitter-vs-plurk/ 

Akbari,  M.  et  al,  2010.  Enabling  communication  and  feedback  in  mass  lectures.  Proc.  IEEE  International  Conference  on 
Advanced  Learning  Technologies  (ICALT  2010),  IEEE  Press,  Sousse,  Tunisia,  pp.  254-258,  doi: 
10.1109/ICALT.2010.76. 

Atif,  Y.,  2012.  Conversational  Learning  Design  in  Future  Classrooms.  Proc.  IEEE  International  Conference  on 
Advanced  Learning  Technologies  (ICALT  2012),  Rome,  Italy,  pp.  178-180,  doi:  10. 1109/ICALT.20 12.49 

Camusi,  A.and  Giovannella,  C.,  2010.  Re-thinking  the  forum.  Proc.  IEEE  International  Conference  on  Advanced 
Learning  Technologies  (ICALT 2010),  Sousse,  Tunisia,  pp.  523-525,  doi:  10.1 109/ICALT.2010.149 

Ebner,  M.,  2013.  The  Influence  of  Twitter  on  the  Academic  Environment.  Patrut,  B.,  Patrut,  M.,  Cmeciu,  C.  (ed.).  Social 
Media  and  the  New  Academic  Environment:  Pedagogical  Challenges.  IGI  Global.  2013.  pp.  293-307 

Fuks,  H.  and  Assis,  R.  L.,  2001.  Facilitating  Perception  on  Virtual  Leamingware-based  Environments.  In  The  Journal  of 
Systems  and  Information  Technology,  Vol.  5,  No.  1,  pp  93-113,  ISSN1328-7265,  Edith  Cowan  University. 

Gerosa,  M.  A.  et  al,  2005.  No  Need  to  Read  Messages  Right  Now:  Helping  Mediators  to  Steer  Educational  Forums  Using 
Statistical  and  Visual  Information.  Proc.  Conference  on  Computer  support  for  collaborative  learning  (CSCL  2005). 
Taipei,  Taiwan,  pp.  160-169. 

Holotescu,  C.  and  Grosseck,  G.,  2010.  Learning  to  microblog  and  microblogging  to  learn.  A case  study  on  learning 
scenarios  in  microblogging  context,  Proc.  International  Scientific  Conference  on  eLearning  and  Software  for 
Education  (eLSE  2010),  Bucharest,  Romania,  pp.  365-374. 

Lama,  M.  et  al,  2011.  Semantic  Linking  of  a Learning  Object  Repository  to  DBpedia.  Proc.  IEEE  International 
Conference  on  Advanced  Learning  Technologies  (ICALT  2011),  IEEE  Press,  Athens,  USA,  pp.  460-464,  doi: 
10.1 109/ICALT.201 1.143 

Lau,  S.  B.  and  Lee,  C.,  2012.  Enhancing  Collaborative  Filtering  of  Learning  Resources  with  Semantically-Enhanced 
Social  Tags.  Proc.  IEEE  International  Conference  on  Advanced  Learning  Technologies  (ICALT  2012),  IEEE  Press, 
Rome,  Italy,  pp.281-285,  doi:  10.1109/ICALT.2012.102 

Lim,  R.,  2008.  Plurk:  A visualized  Pownce  or  another  threaded  twitter?.  Available  at: 

http://plod.popoever.eom/archives/001364.html 

Marques,  A.  M.  et  al,  2013.  Structuring  the  discourse  on  social  networks  for  learning:  Case  studies  on  blogs  and 
microblogs.  In  Computers  in  Human  Behavior,  Vol.  29,  No.  2,  Mar.  2013,  pp.  395-400,  doi: 

10.1016/j.chb.2012.03.001. 

Mendoza,  M.  and  Becerra,  C.,  2010.  On  the  Design  of  Learning  Objects  Classifiers.  Proc.  IEEE  International  Conference 
on  Advanced  Learning  Technologies  (ICALT  2010),  IEEE  Press,  Sousse,  Tunisia,  pp.  464-468,  doi: 
10.1109/ICALT.2010.135 

Muhlpfordt,  M.  and  Wessner.  M.,  2005.  Explicit  referencing  in  chat  supports  collaborative  learning.  Proc.  Conference  on 
Computer  Support  for  Collaborative  Learning  (CSCL  '05),  Taipei,  Taiwan,  pp.  460-469. 

Stepanyan,  K.  et  al,  2010.  A Social  Network  Analysis  Perspective  on  Student  Interaction  Within  the  Twitter 
Microblogging  Environment.  Proc.  IEEE  International  Conference  on  Advanced  Learning  Technologies  (ICALT 
2010),  IEEE  Press,  Sousse,  Tunisia,  pp.  70-72,  doi:  10.1 109/ICALT.2010.27 


198 


lADIS  International  Conference  e-Learning  2013 


INTERACTION  PROBLEMS  ACCESSING  E-LEARNING 
ENVIRONMENTS  IN  MULTI-TOUCH  MOBILE  DEVICES: 
A CASE  STUDY  IN  TELEDUC 


Andre  Constantino  da  Silva’’ Fernanda  Maria  Pereira  Freire^’ Alan  Victor  Pereira  de  Arruda^ 

and  Heloisa  Vieira  da  Rocha"’ 

'institute  of  Computing  (PPG)/UNICAMP,  IFSP,  Campinas,  Brazil  / Hortoldndia,  Brazil 
^NIED/UNICAMP,  Campinas,  Brazil 
'ProFIS/UNICAMP,  Campinas,  Brazil 
"'institute  of  Computing/UNICAMP,  Campinas,  Brazil 


ABSTRACT 

e-Learning  environments  offer  content,  such  text,  audio,  video,  animations,  using  the  Web  infrastructure  and  they  are 
designed  to  users  interacting  with  keyboard,  mouse  and  a medium-sized  screen.  Mobile  devices,  such  as  smartphones  and 
tablets,  have  enough  computation  power  to  render  Web  pages,  allowing  browsing  the  Internet  and  access  e-Leaming 
environments.  But,  their  main  interaction  style  is  touching,  bringing  some  problems,  cosmetics  or  severe  ones.  To 
identify  these  problems  and  correct  them  to  allow  mobile  users  access  the  TelEduc,  an  e-Learning  environment,  we 
planned  a user  test  case  with  four  volunteers  using  smartphones  for  access  it.  The  collected  data  was  analyzed  and  some 
problems  were  identified,  giving  some  insights  about  the  problems  and  barriers  multi-touch  mobile  users  dealt  and  can  be 
easily  corrected  to  give  them  a better  interaction  experience.  To  have  an  e-Learning  environment  with  good  usability  in 
mobile  devices  is  a first  step  to  have  a virtual  learning  environment  for  sharing  data  between  desktop  and  mobile  devices 
and  understand  the  behavior  of  mobile  users  in  this  context. 


KEYWORDS 

Mobile  devices  and  services;  Interfaces,  interactions  and  systems  for  distance  education;  Interface  evaluation;  Usability 
testing  and  evaluation. 


1.  INTRODUCTION 

e-Leaming  environments,  as  Moodle  (2013),  SAKAI  (2013)  and  TelEduc  (2013),  are  Web  applications  with 
tools  to  support  teaching  and  learning  activities  though  the  Web,  taking  advantages  of  the  Web  structure  to 
offer  content  with  text,  images,  audios  and  videos  in  a hypertext  document.  The  environment's  tools  allow 
users  to  create  content,  communicate  with  other  users  and  manage  the  virtual  space,  so  tools  like  chat, 
fomms,  portfolios,  repositories  are  widely  used,  and  tools  that  explore  the  audio  and  video  resource  to 
communication,  such  as  instant  messenger  and  video-conferences,  are  becoming  common  among  the 
environments. 

Mobile  devices,  such  as  smartphones  and  tablets,  are  becoming  increasingly  popular;  most  of  them  have 
touch  screen  displays,  are  easy  to  carry,  have  autonomy  for  hours,  Internet  access  and  enough  computing 
power  to  process  Web  pages.  So,  Web  sites  and  Web  applications,  initially  developed  to  be  used  with 
keyboard,  mouse  and  a medium-sized  display,  are  been  accessed  by  small  touch  screen  devices.  The 
e-learning  environments  fit  in  this  context.  Three  kind  of  solution  to  allow  mobile  users  access  e-Learning 
environment  are  emerging:  specific  device  application,  Web  site  specific  for  mobile  devices,  and 
improvement  of  the  environment  for  mobile  and  desktop  access.  Each  solution  had  vantages  and 
disadvantages.  But,  in  the  case  of  virtual  teaching  and  learning  environments,  we  need  ask:  is  it  fundamental 
have  a specific  environment  for  only  mobile  users?  Since  Web  technology  has  a power  to  integrate  different 
technologies,  we  believe  in  a virtual  teaching  and  learning  environment  that  allow  any  device  accesses. 
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Maurer  et  al.  (2007)  did  some  usability  studies  using  touchscreen  mobile  devices  and  desktop  for 
browsing  in  Web  sites,  shows  that  “more  and  more  people  prefer  using  original  content  instead  of  the  mobile 
version,  especially  for  users  of  new  generation  mobile  devices  like  iPhone  and  Android  phones”.  Another 
result  of  this  work  was  the  users  prefer  to  use  the  standard  Web  site  instead  of  tailored  mobile  versions  of 
Web  site.  But  Schmiedl  et  al.  (2009)  have  a different  opinion;  in  their  research  they  conclude  most  of  the 
users  still  prefer  tailored  versions.  Kaikkonen  (2008)  shows  that  the  standard  and  the  tailored  Web  sites  are 
both  used  but  for  slightly  reasons. 

Since  there  is  not  a clearly  opinion  in  the  researches  about  the  users  preference  in  using  tailored  mobile 
version  or  the  standard  interface,  it  is  important  to  find  and  destroy  barriers  that  mobile  users  meet  when  uses 
standard  Web  interface.  In  e-Learning  context,  which  are  the  barriers  or  the  problems  the  user  dealt?  To 
identify  them  in  TelEduc  environment  we  did  user  tests.  Through  this  work  we  identified  important  design 
issues  to  allow  mobile  users  access  e-Learning  environments  and  Web  applications.  Section  2 presents 
related  work,  Section  3 presents  the  TelEduc  environment.  The  materials  and  method  are  described  in 
Section  4.  The  identified  problems  are  presented  in  Section  5,  and  Section  6 discusses  some  solutions. 
Section  7 presents  the  conclusion  and  future  work. 


2.  RELATED  WORK 

The  e-Leaming  environments"  development  teams  are  building  solutions  to  provide  access  on  mobile 
devices.  Three  kind  of  solution  are  emerging:  specific  device  application;  Web  site  specific  for  mobile 
devices;  and  improve  the  Web  site  for  mobile  and  desktop  access. 

Building  specific  device  application  allows  designing  a suitable  user  interface  for  the  device  and  taking 
advantages  of  smartphone's  features,  such  as  touchscreen  and  camera,  but  needs  develop  an  application  for 
each  mobile  platform,  so  to  be  developed  needs  specific  knowledge  programming  team  and  increases  the 
code  lines  number  to  maintain.  Moodle  community  offers  the  Moodle  App  (2013)  and  Moodbile  (2012),  two 
native  mobile  applications  with  versions  for  the  most  popular  smartphone's  platforms. 

Moodle,  since  version  2.1,  offers  a Web  site  specific  to  mobile  devices,  an  example  for  the  second  type  of 
solutions  for  access  e-Learning  environments  in  mobile  devices.  Building  a specific  Web  site  to  mobile 
device  allows  designing  a suitable  user  interface  for  mobile  devices  taking  account  some  common 
characteristics,  such  small  touchscreen,  but  depends  of  the  browser  to  access  some  platform  features,  such 
GPS,  and  increases  the  code  lines  number  to  maintain  too. 

The  latter  solution  considers  that  smartphones  and  tablets  have  enough  computational  power  to  render 
Web  pages  and  to  do  some  adaptation  if  it  is  necessary,  and  offer  the  same  user  interface  for  any  device.  To 
design  this  kind  of  user  interface  it  is  necessary  to  do  some  usability  studies  to  found  barriers  or  user 
interaction  problems.  Disadvantages  of  this  solution  are  to  depend  of  browsers  to  use  the  mobile  features  and 
the  difficult  of  consider  many  interaction  styles  in  the  same  user  interface. 

This  solution  can  start  from  a user  interface  design  model  for  desktop  and  be  improved  to  consider  mobile 
devices.  So  in  the  initial  design  was  designed  thinking  the  user  will  interact  by  keyboard,  mouse  and  medium 
size  display  and,  allowing  users  accessed  these  applications  on  mobile  devices,  there  is  an  increasing  of 
interaction  styles  number,  such  touchscreen.  With  the  interaction  hardware  changing  the  user  deals  with  new 
interaction  problems.  Shrestha  (2012)  did  user  tests  with  volunteers  using  mobile  devices  equipped  with 
joystick  and  a small  screen  and  points  out  some  problems  when  the  users  try  to  use  mobile  devices  to  do 
specific  tasks  into  Web  sites  designed  to  desktop,  so  the  mobile  Web  browsing  experience  needs  to  be 
improved  to  a more  mobile  friendly  Web  site  and  some  mobile  browser  improvement  (here  we  consider 
browser  as  one  platform  characteristic). 

Considering  only  the  e-Learning  environment  context,  we  agree  to  Maurer  et  al.  (2010)  when  they  argue 
the  user  prefer  to  use  the  standard  version  instead  of  mobile  version  of  Web  site,  and  this  is  one  of  the 
motivations  of  our  work  to  study  the  third  kind  of  solution:  to  access  e-Learning  environment  using  mobile 
devices.  Shrestha  (2012)  studied  the  use  of  joystick  and  a small  screen  to  browsing  into  some  Web  sites, 
while  Maurer  et  al.  (2010)  considers  touchscreen  devices.  All  these  researches  point  out  some  interaction 
problems,  but  the  context  was  not  e-Learning  environments.  In  this  work,  we  identified,  through  users  test, 
problems  when  users  use  touchscreen  mobile  devices  to  access  the  TelEduc  e-Learning  environment. 
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3.  THE  TELEDUC  E-LEARNING  ENVIRONMENT 

The  TelEduc  is  a teaching  and  learning  environment  developed  by  the  Nucleus  of  Applied  Informatics  in 
Education  (NIED)  and  the  Institute  of  Computing  (IC),  State  University  of  Campinas  (UNICAMP),  and 
adopted  in  several  public  and  private  institutions,  like  UNICAMP  through  Ensino  Aberto  project  (Eranco  et 
al,  2003). 

The  TelEduc  environment  was  conceited  in  the  end  of  90,  born  with  the  Cerceau's  Master  dissertation 
(1998),  with  professor  Helolsa  Vieira  da  Rocha  as  advisor,  applying  constructivism  theory  (Papert, 
1986)(Valente,  1993)  in  situated  learning  (Lave  and  Wenger,  1991)  or  in  contextualized  learning  (Valente, 
1999)  for  teacher's  continuance  formation.  In  2001  Eebmary,  the  first  free  version  was  released  over  GNU 
General  Public  License  (GPL),  an  unprecedented  fact  in  the  Brazilian  Educational  Software  scenario.  Many 
public  and  private  institutions  adopted  the  TelEduc  as  platform,  increasing  the  TelEduc  user’s  community, 
and  consequently,  the  development  demand.  This  fact  culminated  in  the  release  of  TelEduc  version  3.0  in 
March  2002.  The  version  3.0  was  completely  redesigned  and  optimized,  reason  for  TelEduc  project  was 
awarded  by  ABED  (Brazilian  Association  for  Distance  Education)  in  the  “Research  and  Development  about 
Distance  Learning”  category.  In  August  2011,  TelEduc  version  4.3  was  released,  with  its  user  interface 
redesigned  to  improve  user  tasks  and  be  more  similar  than  popular  Web  sites. 

TelEduc  is  a system  that  aggregate  administration,  management  and  communication  tools  designed  to 
support  teaching  and  learning  activities.  Some  tools  allow  users  to  create  content,  other  ones  allow 
synchronous  or  asynchronous  communication  among  users,  and  manage  participants  and  courses.  The  course 
page  of  TelEduc  4.3  is  structured  in  two  parts:  the  left  one  (Eig.  la)  has  a list  of  all  tools  available  and  in  the 
right  one  (Eig.  lb)  the  content  of  the  selected  tool. 

In  the  course  showed  at  Eig.  la,  a print  screen  of  the  course  made  for  our  user  tests  (the  content  is  in 
Portuguese),  the  teacher  dispose  the  Course  Dynamic,  Agenda,  Activities,  Support  Material,  Readings, 
Erequently  Asked  Questions,  Polls,  Required  Stop,  Bulletin  Board,  Discussion  Eorums,  Chat,  Mail,  Profile, 
Portfolio,  Access,  Intermap  and  Search  tools.  Agenda  is  the  ongoing  course  home  page  and  shows  the 
course's  program  for  a given  period  (daily,  weekly,  etc.).  Agenda  is  an  important  tool  because  organize  the 
activities  that  must  be  done  in  a specific  period,  similar  teachers  do  in  the  beginning  of  a presence  class.  Pig. 
lb  shows  the  Agenda  tool  visualized  by  a student,  where  the  student  can  read  the  agenda  content  and  access 
the  previous  agendas. 


(a)  (b) 

Figure  1 . A desktop  print  screen  of  TelEduc  course  page  (a)  tool  menu  and  (b)  tool  content. 
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The  Readings,  Activities,  Frequently  Asked  Questions  and  Support  Material  tools  have  similar  features 
and  user  interface,  but  different  purposes.  Readings  tool  is  used  to  publish  relevant  documents,  like  books, 
magazines,  news  and  articles.  The  Activities  tool  is  an  area  to  publish  activities  to  the  accomplished  during 
the  course,  like  home  work  descriptions.  The  Frequently  Asked  Questions  tool  contains  a list  of  the  most 
frequently  questions  done  by  the  participants  during  the  course  and  their  respective  answers.  Support 
Material  offers  a virtual  space  for  teachers  publish  useful  information  about  the  course  and  the  proposed 
activities. 

Tools  like  Discussion  Forums  and  Mail  are  used  to  participant  communication,  supporting  text  message 
exchange  in  asynchronous  mode.  To  synchronous  communication,  there  is  the  Chat  tool,  its  features  is 
similar  to  Web  chat  sites. 

The  Portfolio  is  a communication  tool  that  aims  to  promote  the  collaboration  among  participants  through 
the  sharing  of  “items”  (documents,  presentations,  programs,  links,  etc.).  So  the  Portfolio  tool  provides  an  area 
to  item  storage  and  sharing  for  each  participant  (user  or  group  of  users)  within  a course.  The  Bulletin  Board 
tool  is  a dedicated  space  where  all  the  participants  can  post  information  considered  relevant  to  the  course 
content. 

To  provide  content,  TelEduc  uses  the  Web  infrastructure,  more  specifically,  hypertext  with  images,  links, 
audios  and  videos.  All  these  media  can  be  published  as  content  in  tools  like  Agenda,  Support  Material  and 
Readings.  To  create  text  content,  the  user  interacts  with  a rich  text  editor,  the  CKEditor  (2013).  CKEditor  is  a 
third-party  WYSIWYG  text  editor  to  be  used  inside  Web  pages,  bringing  to  the  Web  application  common 
editing  features  found  on  desktop  editing  text  applications,  but  CKEditor  version  3.3  does  not  work  on 
mobile  devices.  In  previous  work  (Da  Silva,  & da  Rocha,  2012),  we  studied  some  problems  in  use  of  third- 
party  software  on  TelEduc  to  create  and  to  visualize  documents,  describing  that  mobile  compatibility  needs 
to  be  considered  to  not  prejudice  mobile  users. 

The  Agenda,  Activities,  Support  Materials,  Readings,  Mail  and  Portfolio  tools  allow  attaching  files  into 
them  items  in  similar  way:  the  user  clicks  in  the  “Attach  File”  button,  select  the  file  to  be  attached  using  a 
dialog  and,  after  the  click  on  the  save  button,  the  file  is  uploaded. 

Since  the  e-Learning  environments  need  to  be  easy  to  use  for  users  with  different  levels  of  Web 
experience,  the  usability  is  an  important  nonfunctional  requirement.  TelEduc  was  designed  in  an  iterative 
design-evaluation  process  to  have  good  usability  and  the  user  interface  does  not  impair  teaching  and  learning 
activities,  so  many  usability  evaluations  were  done.  The  accessibility  is  another  nonfunctional  requirement 
desired  for  TelEduc,  to  allow  impaired  people  to  use  the  environment  without  meet  barriers  or  obstacles. 

TelEduc  was  designed  to  use  a mouse  and  keyboard  as  input  devices,  and  a medium  screen  size  as  output 
device.  TelEduc  is  better  visualized  into  1024x200  pixels  screen  resolution.  Visualize  it  into  a lower  screen 
resolution  cause  some  user  interface  problems  like  dispose  interface  components  in  wrong  position.  Beside 
the  problems  happen  due  the  resolution  changing,  which  problems  and  barriers  the  mobile  users  met  when 
interact  with  TelEduc?  To  answer  this  question,  we  developed  a user  test  case  described  in  the  next  section. 


4.  MATERIALS  AND  METHOD 

To  identify  interaction  problems  when  an  e-Learning  environment  is  accessed  by  a mobile  device  we  planned 
a study  case  composes  with  four  users  tests.  The  main  activities  to  do  the  study  case  were:  (i)  tools  to  be 
investigated  and  tasks  definition;  (ii)  hardware  to  be  used;  (iii)  questionnaires  planning;  (iv)  preparation  of 
the  course. 

The  following  sub-sections  explain  briefly  each  activity.  We  started  choosing  the  tools  to  be  evaluated 
and  defined  the  tasks  for  user  test  sessions  using  the  smartphones  available,  an  Android-based  and  an  iPhone. 

Based  on  our  experience  in  using  e-Learning  environments,  we  chose  the  tools:  Agenda,  Activities, 
Readings,  Polls,  Mail  and  Portfolio.  Eor  each  tool  was  defined  one  task,  considering  the  most  important 
feature  of  the  tool.  All  tools  were  used  considering  the  student's  view.  The  defined  tasks  were:  1.  Login  in 
TelEduc;  2.  Read  the  Agenda;  3.  Read  the  Reading  and  do  it;  4.  Read  the  Activity;  5.  Post  her  activity  in  her 
portfolio;  6.  Answer  a poll;  7.  Send  e-mail  to  the  teacher;  8.  Logout  of  TelEduc. 

To  do  the  task  5,  the  volunteers  need  to  search  into  the  Web  for  an  article.  The  purposeful  activity  is  to 
allow  us  identify  interaction  problems  when  the  users  need  to  navigate  in  two  web  pages,  visualizing  one 
page  and  going  to  another.  In  the  tasks  planning  we  found  some  interaction  problems  that  impair  the  user  to 
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conclude  the  tasks,  so  we  shaped  the  defined  activities  such  a way  the  user  does  not  meet  the  identified 
problems  with  the  goal  to  find  new  interaction  problems.  The  problems  identified  on  task  planning  are 
discussed  in  Section  5.1. 

We  used  a Motorola  Milestone  smartphone  (Motorola  Mobility,  2013)  and  an  Apple  iPhone  3GS  (Apple, 
2013).  The  Motorola  Milestone  has  a 3.7  inches  multi-touch  display  with  133MB  internal  storage  memory 
expansive  up  to  32  GB  with  a memory  card,  600  MHz  Cortex-A8  processor  and  256  MB  RAM,  a 5MP 
camera,  GPS  and  wireless  connection  by  Wi-Fi  802.11  b/g  and  bluetooth.  Android  4.0.3  (Google,  2013)  was 
used  as  operation  system  (OS).  Motorola  Milestone  has  a proximity  sensor,  an  ambient  light  sensor,  a 3 -axis 
accelerometer  and  a geomagnetism  sensor  to  provide  orientation  with  respect  to  Earth's  magnetic  field.  To 
browse  in  the  Web  application,  the  volunteers  use  the  Android  stock  Web  browser. 

The  Apple  iPhone  3GS  (Apple,  2013a)  has  a 3.5  inches  multi-touch  display  with  32  Gb  internal  storage 
memory,  600  MHz  Cortex-A8  processor  and  256  MB  eRAM,  a 3MP  camera,  GPS  and  wireless  connection 
by  Wi-Fi  802.11  b/g  and  bluetooth.  The  installed  operation  system  is  iOS  version  5 (Apple,  2013b).  iPhone 
has  too  a proximity  sensor,  an  ambient  light  sensor,  a 3-axis  accelerometer  and  a geomagnetism  sensor.  To 
browse  in  the  Web  application,  the  volunteers  use  the  Safari  browser.  In  both  devices,  the  proximity  sensor 
and  the  accelerometer  can  be  used  to  interact  with  applications,  but  the  browsers  do  not  use  these  features  as 
input.  Only  the  touchscreen  was  used  as  input  device. 

To  collect  information  about  the  volunteers  and  them  opinion  we  elaborated  two  questionnaires.  The  first 
one  was  to  collect  data  about  user  age,  genre,  internet  usage  and  mobile  skills.  The  second  one  was  to  get 
personal  satisfaction  in  do  the  tasks.  Due  we  did  an  experiment  involving  people,  we  prepared  a consent  form 
to  be  signed  by  the  volunteers  informing  her  about  the  study  purposes,  the  anonymity  of  the  volunteer,  and 
allowing  recording  the  interaction  to  further  analysis. 

After  the  task  planning,  we  prepared  a course  in  the  TelEduc  installed  in  our  server  with  the  agenda 
content,  the  activity  description,  an  article  published  on  Readings  tool,  and  the  poll  to  be  answered.  This 
preparation  is  to  simulate  a course  in  the  environment  with  chained  activities.  Since  the  user  tests  laboratory 
and  the  server  share  the  same  local  network,  we  did  not  worry  about  Internet  lacks  or  fail. 

We  invited  four  undergraduate  students  to  participate,  one  man  and  three  women,  whom  filled  the  profile 
questionnaire  before  the  participation.  All  volunteers  have  20  years  old,  three  of  them  have  smartphone  and  is 
mobile  users  for  more  than  six  month,  the  last  one  does  not  have  a smartphone  and  her  consider  not  be  a 
mobile  user. 


5.  DATA  ANALYSIS 

In  this  section  we  discuss  about  the  found  problems  when  mobile  users  access  the  TelEduc  Environment.  We 
found  one  problem  during  the  tasks  definition,  and  discuss  it  here  since  impose  a barrier  and  do  not  allow 
mobile  users  conclude  their  goal.  Analyzing  the  user  tests  data,  we  found  5 relevant  problems  for  mobile 
users  and  other  usability  problems  that  is  device  independently.  We  focused  here  in  the  mobile  problems  and 
in  the  end  of  the  section  we  present  solutions. 

5.1  Problem  found  in  the  Task  Planning 

TelEduc  uses  CKEditor  to  allow  users  write  rich  text  instead  of  simple  plain  text,  but  CKEditor  does  not 
work  in  iPhone  and  in  Android-based  devices,  bringing  a barrier  to  user  write  rich  text.  For  example,  it  is  not 
possible  put  content  in  a Portfolio  item  because  the  content  field  does  not  appear  (Problem  1).  Looking  in  the 
environment  for  others  barriers  caused  by  CKEditor,  we  found  that  it  is  possible  to  write  message  in  Mail 
tool  and  post  Bulletin  Board  posts,  but  the  content  field  appears  to  write  simple  text  since  CKEditor  is  not 
displayed.  In  the  other  environment  tools  the  barrier  was  found. 

Focused  on  discovery  new  interaction  problems  we  planned  the  tasks  such  a way  the  users  do  not  match 
this  problem.  The  next  sub-section  presents  the  problems  discovery  through  the  user  tests. 
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5.2  Problems  found  in  the  User  Tests 

The  first  identified  problem  in  user  tests  {Problem  2)  is  related  with  the  web  site  does  not  follow  the 
smartphone  interface  standard.  The  first  time  the  user  2 tried  to  go  to  another  tool,  she  touched  in  the  space 
between  the  wanted  menu  item  the  symbol  ‘>’  (Fig.  2a  shows  the  point  tapped  by  the  user  in  the  interface), 
since  no  action  was  triggered,  she  tried  a lot  of  times  again  in  the  same  place.  Asked  about  the  unexpected 
system  response,  the  user  touched  in  the  tool  label  and  the  system  shows  the  tool  content.  Maybe  the  user 
thought  the  TelEduc  interface  follows  the  smartphones  standard,  which  is  possible  to  trigger  an  action 
touching  in  the  space  nearby  the  menu  item  label  (Fig.  2b).  In  desktops  it  is  softened  due  to  change  the  mouse 
cursor  from  an  arrow  to  a pointing  hand,  giving  a visual  clue  that  it  does  not  have  in  touchscreen  mobile. 


Course  Dynamic  > 

Agenda 

Activities  ^ 

Support  Material  > 

Readings  > 

Frequently  Asked  Questions  > 
Polls  > 

Required  Stop 

Bulletin  Board  > 


Discussion  Forums  > 


(a) 


(b) 


Figure  2.  Print  screens  marked  with  the  place  tapped  by  the  user  in  (a)  TelEduc  tool  menu  and  no  action  was 
triaeered  and  tbl  in  iPhone  menu  settines.  where  an  action  was  trieaered. 

Another  problem  observed  during  the  user  tests  was  related  with  the  selection  function  {Problem  3). 
Three  users  needed  help  to  select  the  desirable  URL  to  perform  the  copy  and  paste  functions.  Maybe  due  the 
users  are  familiar  with  desktops,  they  were  expected  for  a menu  or  a toolbar  with  options.  Asked  about  how 
they  interact  with  the  device,  they  realized  about  the  touch  interface  and  they  tried  to  tap  the  desirable  text, 
and,  in  a second  try,  tapped  the  text  and  hold  the  finger  over  the  text,  and  the  interface  responses  with  the 
options.  The  problem  is  about  the  visibility  and  discoverability  of  the  actions.  We  do  not  realize  that  we  will 
meet  this  problem,  since  the  all  volunteers  are  the  generation  who  use  mobile  devices  since  childhood,  but 
they  have  a mental  model  more  related  with  the  desktop  computers  than  mobile  devices. 

After  copy  the  URL  to  do  the  defined  activity  in  task  4,  two  users  had  another  problem:  how  to  return  to 
TelEduc,  since  another  tab  was  opened  {Problem  4).  The  button  to  trigger  the  action  to  show  opened  tabs  is 
at  bottom  on  iPhone  and  is  in  at  top  on  Android-based  smartphones.  One  of  the  users  prefers  typing  the 
TelEduc  address  again  instead  of  searching  for  how  to  go  to  an  opened  tab.  The  other  user  asked  for  help. 

Most  of  TelEduc  feedback  appears  on  the  page  top  (Fig.  3a),  and  to  draw  the  user's  attention  the  feedback 
blinks.  In  mobile  devices,  with  the  user  is  visualizing  another  part,  the  feedback  is  not  visible  {Problem  5). 
One  example  happens  when  the  user  is  posting  a URL  in  the  Portfolio  item  (doing  the  task  5),  and  after  to  tap 
the  “Ok”  button,  the  system  response  with  the  feedback  “Internet  Addresses  included  successfully.”  in  the 
page  top,  but  the  user  does  not  see  since  she  was  interaction  with  the  elements  in  the  bottom  (Fig.  3b).  It  is 
important  to  highlight  that  the  TelEduc  form  to  show  feedbacks  differs  from  the  smartphones  form  to  do  it.  In 
smartphones  feedbacks  were  showed  as  a popup  dialog. 

The  last  identified  problem  is  about  the  time  a feedback  message  is  displayed  {Problem  6).  When  the  user 
sends  a message,  the  TelEduc  shows  a feedback  page  to  inform  that  the  message  was  successful  sent  for  2000 
milliseconds.  Since  the  smartphones  take  more  time  to  render  the  page  and  shows  all  the  page  content,  the 
waiting  time  expires;  the  browser  closes  the  feedback  page  almost  immediately,  the  user  can  read  the 
feedback  message,  giving  the  impression  that  an  error  happed. 

Analyzing  the  user  opinion  questionnaire  and  considering  all  identified  problems,  the  users  agree  with 
TelEduc  have  good  usability.  Three  of  the  four  volunteers  described  feel  difficult  to  interact,  and  the  last  one 
related  does  not  feel  any  difficult  to  interact. 
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5.3  Solutions 


(a) 


(b) 


Figure  3.  Print  screens  of  TelEduc  feedback  on  (a)  desktop  and  (b)  mobile  devices  after  the  user  click  in  the  “Ok” 
link  to  post  the  link  in  the  Portfolio  item  to  do  the  task  5. 

Problem  1 is  about  the  mobile  user  cannot  write  the  content  in  most  of  TelEduc  due  the  CKEditor  is  not 
displayed  on  mobile  devices.  A good  solution  is  to  allow  mobile  user  only  write  plain  text,  similar  happen  to 
write  messages  in  TelEduc  Mail  Tool.  Analyzing  the  Portfolio  tool  code,  we  discovery  that  the  content  field, 
a textarea  tag,  was  not  showed  due  a style  property  (displaymone)  that  CKEditor  changes  when  it  loads. 
Since  mobile  devices  do  not  load  the  CKEditor,  this  property  is  not  changed.  Changing  the  code  to  get  the 
content  from  the  value  property  when  CKeditor  is  not  load,  and  get  the  content  from  CKEditor  when  it 
loaded,  resolve  the  problem,  destroying  this  barrier. 

Problem  2 (user  tapping  in  the  space  between  menu  item  label  and  the  symbol  ‘>’  to  trigger  an  action)  is 
easily  to  resolve.  The  tool  menu  is  implemented  with  table  and  anchor  tags,  just  changing  the  order  putting 
the  li  tag  inside  the  <a>  tag,  becoming  all  the  cell  area  a link. 

Problem  3 (trigger  a text  selection)  and  Problem  4 (return  to  a previous  tab)  are  related  with  the 
smartphones  operation  system.  Norman  and  Nielsen  (2010)  discuss  discoverability  and  visibility  problems  in 
touchscreen  devices,  and  they  argue  there  is  a lack  of  established  guidelines  for  gestural  control. 

A solution  for  the  Problem  5 (TelEduc  feedback  is  fixed  at  the  top)  is  show  feedback  messages  using 
JavaScript  dialog  when  the  device  have  JavaScript  support.  Since  browsers  for  mobile  devices  have 
JavaScript  engine  inside,  the  browser  have  the  responsibility  to  show  the  message  in  the  visual  standard  used 
in  the  mobile  device.  This  solution  may  be  applied  to  the  Problem  6 (feedback  time),  giving  the  user  the 
feeling  of  control  over  the  application. 


6.  CONCLUSION  AND  FUTURE  WORK 

Mobile  devices,  such  as  smartphones  and  tablets,  are  becoming  increasingly  popular;  most  of  them  have 
touch  screen  displays,  Internet  access  and  enough  computing  power  to  process  Web  pages,  allowing  mobile 
users  access  Web  applications,  such  e-Learning  environments.  The  e-Learning  environments  were  initially 
designed  to  be  used  with  keyboard,  mouse  and  a medium  size  display,  and  now  have  to  consider  their  access 
in  mobile  devices.  We  believe  that  Internet  is  a meeting  point  for  all  devices,  so  the  Web  applications  need  to 
be  access  by  any  device.  Which  kind  of  barriers  and  problems  do  mobile  users  meet  when  browsing  in  an 
e-Leaming  environment?  We  did  a user  test  case  with  four  volunteers  accessing  the  TelEduc  environment 
through  iPhone  and  Android-based  smartphones,  and  discovery  6 problems  that  impact  in  the  usability.  Do 
usability  studies  with  users  is  a hard  work,  but  it  is  necessary  due  the  lack  of  techniques  to  do  this  kind  of 
evaluation.  The  discovered  problem  is  related  with  the  consistence  between  device  interfaces  and  TelEduc 
interface,  discoverability  and  visibility  of  the  features  and  code  structure. 

e-Leaming  environments  have  a lot  of  contributions  to  m-Learning,  offering  a platform  to  teachers  and 
students  do  their  teaching  and  learning  activities,  but  some  technologic  gaps  need  to  be  filled.  With  this  work, 
we  started  with  the  first  step  finding  and  eliminating  barriers  and  interaction  problems  that  become  hard 
mobile  users  access  e-Learning  features.  Another  step  is  taking  advantages  of  the  mobile  device  features  in 
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pro  learning  activities  such  create  content  like  photos  and  videos.  More  research  is  need,  since  the  interaction 
style  changed  and  different  features  are  available,  e.g.,  the  mobile  users  write  text  with  fewer  words? 
Analyzing  the  mail  message  sent  from  our  volunteers  to  teachers  to  inform  the  activity  was  done,  we  noted 
just  a sentence  was  write.  Is  it  happened  due  to  the  profile  of  our  younger  volunteers  or  due  the  mobile  device 
restrictions?  How  this  influences  in  the  activities  in  the  environments?  Mobile  users  prefer  record  a video  or 
write  a big  text?  These  questions  needs  more  research  to  be  explained,  but  needs  an  e-Learning  environment 
without  barrier  to  be  accessed  on  mobile  devices. 
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ABSTRACT 

The  integration  of  different  platforms  and  information  Systems  in  the  academic  environment  is  highly  important  and 
quite  a challenge  within  the  field  of  Information  Technology.  This  integration  allows  for  higher  resource  availability  and 
improved  interaction  among  intervening  actors.  In  the  field  of  e-Learning,  where  Learning  Management  Systems  are  used 
to  develop  the  courses,  their  integration  with  other  platforms  and  applications  is  more  than  interesting,  in  particular  with 
Digital  Repositories.  This  article  describes  the  integration  of  the  Moodle  LMS,  used  by  the  School  for  the  last  seven 
years,  with  the  DSpace  repository,  currently  widely  spread  as  well.  The  integration  process  consists  of  two  stages,  in 
order  to  establish  full  communication  to  and  from  the  repository.  For  communication  from  Moodle  to  the  repository,  with 
the  goal  of  consulting  and  transferring  elements  from  DSpace,  modules  already  implemented  for  both  platforms  were 
used.  The  current  stage  is  implementing  a specific  module  in  order  to  establish  communication  in  the  other  direction  and 
allow  transferring  resources  and  learning  objects  from  the  LMS  to  the  repository.  This  module  is  very  useful  for 
educational  material  that  can  be  made  publicly  available  through  a repository  and  thus  transcending  the  borders  of  an 
educational  environment.  The  initial  implementation  is  oriented  towards  the  publication  of  work  done  by  the  students  and 
presented  for  evaluation  through  the  Moodle  platform.  The  incorporation  of  metadata  to  the  published  resource  is 
automatic,  taking  context  information  from  the  platform,  and  at  a later  stage,  with  the  intervention  of  librarians,  who 
contribute  their  knowledge  in  the  classification. 


KEYWORDS 

E-learning,  LMS,  repositories.  Learning  Objects 


1.  INTRODUCTION 

In  current  times,  information  systems  integration  is  paramount,  and  constitutes  a challenge  in  the  field  of 
Information  Technology.  Systems  are  generally  not  designed  for  integration.  Instead,  they  are  typically 
developed  to  solve  specific  problems.  Thus,  each  system  uses  languages  and  technologies  of  its  own, 
according  to  the  needs  of  each  application,  which  makes  communication  between  them  difficult.  Integration 
comes  from  the  need  to  share  data  among  heterogeneous  systems,  in  order  to  achieve  unification  in 
information  access  and  create  the  impression  in  users  that  they  are  interacting  with  a single  system.  An 
integral  view  allows  for  the  retrieval  and  reuse  of  information  through  one  single  point  of  access. 

There  is  a wide  set  of  applications  in  multiple  environments  that  benefit  from  information  integration,  for 
example,  in  the  area  of  e-commerce  and  e-business,  business  services  and  transactions  can  be  made  simple 
through  networks  [1].  In  the  field  of  e-Learning,  where  Learning  Management  Systems  (LMSs)  are  used  for 
course  development,  their  integration  with  other  platforms  and  applications  can  help  increase  resource 
availability  and  communication  among  users.  In  this  field.  Learning  Objects  (LOs)  are  the  basis  of 
interoperability,  which  is  why  integration  among  intervening  systems  is  particularly  important  in  this  context. 
The  possibility  of  integrating  LMSs  with  general  purpose  Digital  Repositories  and  Learning  Object 
Repositories  (LORs)  proposes  a change  in  how  teachers  think,  plan  and  build  educational  content  [2]. 
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The  Moodle  LMS  has  been  used  in  the  School  for  over  seven  years.  There  is  a great  amount  of  stored 
academic  material  and  a high  number  of  users.  Currently,  in  light  of  the  new  trends  in  educational  material 
and  with  the  growth  of  OERs  (Open  Educational  Resources),  there  arose  the  initiative  of  storing  content  in 
one  single  public  space  available  for  everyone.  Eor  this  reason,  DSpace  began  to  be  used  as  a repository  to 
store  LOs  and  all  the  content  produced  by  the  School,  in  order  to  facilitate  its  retrieval  and  reuse. 

This  article  describes  the  integration  of  the  Moodle  LMS  with  the  DSpace  repository,  both  from  and  to 
the  repository.  It  will  describe  in  detail  the  implementation  of  the  module  that  will  allow  communication 
towards  DSpace  for  the  transference  of  learning  objects  from  the  LMS  to  the  repository.  The  transference 
from  DSpace  to  Moodle  was  solved  through  the  Rest  API  provided  by  DSpace  and  a specific  module 
available  in  Moodle  [3]. 

The  first  stage  consists  of  the  development  of  the  module  that  allows  communication  from  Moodle  to 
DSpace  for  a particular  use  case  that  is  the  publication  of  work  done  by  the  students  and  stored  in  the  LMS 
through  the  Tasks  functionality  in  Moodle.  This  paper  will  describe  the  way  in  which  the  resource  is  prepared 
for  its  publication  and  the  process  of  incorporation  of  the  metadata.  In  order  to  show  the  work  context,  we 
will  also  describe  the  use  of  the  LMS  in  the  School  as  a fundamental  e-Learning  tool  for  the  dissemination  of 
educational  material  and  the  advantages  of  the  use  of  a repository  for  storing  the  resources  generated  within 
the  institution. 


2.  DSPACE  DIGITAL  REPOSITORY 

The  current  academic  and  scientific  development  of  all  universities  increases  the  general  production  of 
scientific  material  and  consequently  generates  a need  to  group,  store,  preserve  and  distribute  great  amounts  of 
information  in  a timely  manner.  Thus,  it  gives  rise  to  new  ways  to  store  information,  such  as  the  digital 
library  at  first,  and  later,  the  digital  repository,  a more  global  and  abstract  method.  Digital  repositories  are 
virtual  spaces  that  store  sets  of  documents  in  multiple  digital  formats,  organized  according  to  preestablished 
criteria  [4]. 

There  are  currently  a number  of  initiatives  that  develop  and  encourage  the  use  of  repositories  in  general 
and  LORs  in  particular,  a fact  that  can  be  verified  by  checking  the  repositories  that  are  available  online,  with 
tens  of  thousands  of  resources  in  storage.  There  are  also  initiatives  that  are  working  towards  interoperability 
between  repositories,  with  the  goal  of  generating  networks  of  distributed  systems  for  general  searches  [5]. 

In  our  case,  the  institutional  repository  was  created  out  of  a need  to  store  in  a single  publicly  accessible 
location  the  material  produced  by  the  School  that  deserves  global  distribution.  Consequently,  a repository 
was  implemented  through  DSpace,  an  open  source  application  that  offers  a wide  range  of  functionality  for 
resource  management.  The  first  stage  will  consist  of  the  storage  of  projects  developed  by  students  for  one  of 
the  subjects  of  the  School.  These  projects  are  sometimes  assigned  by  the  teachers,  or  written  by  students  who 
become  especially  interested  in  a certain  topic  and  choose  to  develop  a related  application. 

If  the  subject  contemplates  this  option,  the  best  projects  can  be  used  with  real  users  to  meet  specific 
demands,  such  as,  for  example,  a system  that  kept  the  statistics  of  a local  soccer  team,  or  one  that  kept  a 
record  of  the  resources  in  the  historical  archive  of  the  Province  of  Buenos  Aires,  to  name  a few  examples. 
Later  stages  contemplate  the  incorporation  of  learning  objects  designed  for  courses,  final  reports,  and  degree 
theses. 

In  order  to  store  the  aforementioned  educational  material,  a structure  of  collections  will  be  created  and 
divided  into  subjects.  The  resources  will  be  classified  using  descriptive  metadata  related  to  the  characteristics 
of  the  project,  such  as  topic,  platform  used,  and  installation  requirements,  among  others.  The  possibility  of 
permanently  and  publicly  storing  these  projects  will  allow  other  students  to  become  acquainted  with  them 
and  will  serve  as  a basis  for  their  further  development.  These  developments  can  be  also  used  by  the  teachers 
of  other  subjects  aiding  as  examples  of  applications  and  technologies  in  use. 

The  repository  that  was  implemented  at  the  School,  DSpace  version  1.8,  is  currently  beta  in  design  and 
content,  and  can  be  found  at  http://dspace.linti.unlp.edu.ar. 


208 


lADIS  International  Conference  e-Learning  2013 


3.  MOODLE  VIRTUAL  PLATFORM 

The  School  has  been  working  with  the  Moodle  virtual  platform  for  online  course  management  for  over  7 
years  as  a complement  to  in-person  undergraduate  and  postgraduate  classes,  as  well  as  courses  offered  by  the 
Secretary  of  Extension.  The  platform  includes  over  10000  registered  users  in  around  170  courses. 

The  study  material  is  comprised  of  digital  resources  in  multiple  formats.  For  example,  some  Extension 
courses  have  experimented  with  the  use  of  standardized  virtual  classes  through  learning  objects.  The 
possibility  of  using  LOs  that  comply  with  the  SCORM  standard  allows  not  only  reusing  them,  but  also 
following  and  keeping  a record  of  the  progress  of  each  student.  All  the  content  is  also  published  in  HTML, 
PDF  and  slideshow  format.  Some  cases  include  videos  that  work  as  triggers  for  new  topics  or  as 
complements  to  an  explanation.  These  videos  are  published  in  the  platform  and  linked  through  the  resource 
of  a website.  Software  is  also  included  when  necessary  in  the  proposed  activities,  as  well  as  links  with  extra 
contents  and  ad-hoc  specific  material  for  web  developers  [6]  [7].  Fig.  1 shows  the  used  of  multiple  resources 
in  the  platform  through  the  years. 
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Figure  1 . Resources  used  in  the  Moodle  virtual  platform 

Content  distribution,  both  theoretical  and  practical,  is  achieved  mostly  through  files  in  multiple  formats 
such  as  PDFs,  text,  and  slides,  among  others.  These  files,  together  with  tags  for  course  organization  and 
URLs  for  further  information,  make  up  the  three  most  used  types  of  resource  in  the  virtual  platform. 

Activities  include  discussion  forums,  used  for  communication  between  students  and  teachers,  and  in 
some  cases  for  general  notifications,  such  as  exam  dates  or  other  important  information.  Questionnaires 
have  been  used  in  many  subjects  for  approximately  two  years,  and  are  very  useful  for  self  evaluations  or 
discussions  of  specific  topics,  in  mass  courses.  Graduate  courses  tend  to  use  them  before  virtual  encounters  in 
order  to  level  contents  so  as  to  better  seize  them.  Choices  are  used,  for  example,  to  survey  the  general 
opinion  of  the  students  on  an  activity  or  a class  or  specific  administrative  matters,  or  even  to  measure 
attendance  to  a mid-term.  Fig.  2 shows  the  use  of  the  activities  in  the  courses.  As  seen,  discussion  forums  are 
the  most  used,  mainly  for  communication  with  the  students,  both  in  academic  and  administrative  matters. 
Assignments  come  second  and  are  the  central  motivating  axis  of  this  paper.  This  workgroup  has  also 
developed  a social  networking  connection  module,  a Twitter  Activity  Module 
[https://github.com/mcharnelli/moodle-module_twitter],  growing  in  use  for  communication  between  students 
and  teachers. 
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Figure  2.  Activities  used  in  the  Moodle  virtual  platform 

Assignment  is  one  of  the  most  used  modules  because  of  the  practical  nature  of  the  courses  that  habitually 
use  the  virtual  platform.  Through  assignments,  students  can  upload  files  that  can  later  be  evaluated  by  their 
teachers,  who  grade  and  give  feedback  through  Moodle.  Grade  and  feedback  are  sent  to  students  by  email 
and  can  only  be  seen  by  them  and  the  teachers.  This  is  useful  both  for  teachers  and  students,  as  the  latter  can 
upload  their  work  online  from  home  and  any  time  before  the  established  deadline. 

As  previously  mentioned,  some  projects  are  developed  with  a specific  goal  in  mind  and  for  use  in  one 
institution  that  requires  them.  They  are  also  sometimes  used  as  material  for  the  course  in  the  years  that 
follow,  although  these  are  rare  cases  and  generally  discarded  each  year.  Taking  the  amount  of  total  projects 
uploaded  to  the  platform  into  account,  which  is  over  2700,  the  waste  is  significant  and  there  is  high  reuse  and 
resignification  potential  within  the  same  course  or  by  other  educational  institutions  that  find  it  useful. 


4.  COMMUNICATION  BETWEEN  DSPACE  AND  MOODLE  THROUGH 
THE  SWORD  PROTOCOL 

As  mentioned  in  previous  sections,  both  the  Moodle  LMS  and  the  DSpace  repository  use  their  own 
technologies  according  to  their  needs,  and  in  many  cases  their  integration  is  not  immediate,  despite  the 
increasing  flexibilities  they  have  been  offering  through  time. 

The  development  of  the  Moodle  platform  has  been  continuously  growing.  Moodle  Version  2.0 
incorporated  the  concept  of  repository  and  communication  with  other  well  known  repositories  such  as 
Alfresco,  Merlot,  Flicker,  Google  Drive,  Picasa,  and  Dropbox  was  implemented.  This  feature  considerably 
increased  flexibility  in  file  handing  and  access  to  external  repositories  within  the  platform  itself.  Although 
this  extension  is  very  useful,  it  is  important  to  highlight  that  communication  is  always  established  in  one 
direction  only,  from  Moodle. 

The  initial  goal  was  to  obtain  communication  from  Moodle  to  the  repository,  with  the  goal  of  retrieving 
and  transfering  elements  from  DSpace.  Modules  were  used  to  extend  the  functionality  of  the  platforms  that 
would  be  involved  in  the  new  communication  channel.  In  the  repository,  a module  that  installs  an  API  using 
REST  and  that  is  available  through  the  DSpace  community  was  used.  The  LMS  has  a Repository  API  that 
was  used  to  incorporate  a specific  plugin  to  DSpace,  created  by  the  Moodle  community. 

In  relation  to  the  other  communication  channel,  a new  functionality  was  implemented  that  makes  it 
possible  to  publish  content  from  the  Moodle  platform  to  different  repositories.  This  new  functionality  is  very 
useful  to  export  all  kinds  of  material  from  the  courses,  such  as  theoretical  or  practical  content, 
documentation,  and  projects  delivered  by  the  students.  It  is  also  possible  to  export  standardized  content  as 
learning  objects. 
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The  development  of  this  tool  involved  the  Sword  protocol  [9],  which  implements  simple  remote  storage 
of  resources  into  a repository  from  other  applications. 

In  our  application  case,  the  functionality  was  incorporated  from  the  interface  of  the  Assignment  module 
in  Moodle,  which  allows  for  automatic  delivery  of  these  assignments.  Fig.  3 shows  the  send  to  repository  link 
(Enviar  al  repositorio)  on  the  upper  left  corner,  circled  in  green.  This  link  will  establish  communication 
between  the  LMS  and  the  repository,  described  in  detail  in  later  sections. 


Ver  todas  las  calificaciones  del  curso 

Nombre  i TodosABCDEFGHIJKLMNNOPQRSTUVWXYZ 
Apellido  ; TodosABCDEFGHIJKLMNNOPQRSTUVWXYZ 


Enviar  al  Repositorio 

Descargar  todas  las  tareas  en  un  zip 

Nombre  / Apellido  4’ 

Direccion  de  correo  Calificacion  Comentario 

ultima  modificacion  (Envio)  Ultim 

Paola  Amadeo 

pamadeo@gmail.com 

S pamadeo.zip 

jueves.  20  de  diciembre  de 
2012.  11:27 

Marfa  Emilia  Charnelli 

mecharnelli@gmail.com 

Alejandra  Schiavoni 

aschiavoni@gmail.com 

S aschiavoni.zip 

jueves.  20  de  diciembre  de 
2012.  11:29 

Figure  3.  Interface  for  the  visualization  of  Moodle  tasks,  with  the  Send  to  Repository  link 


4.1  Metadata  Assignment 


The  incorporation  of  resources  into  a repository  requires  the  generation  and  instantiation  of  metadata  that  act 
as  indices  to  locate  technological  resources.  Making  use  of  the  right  metadata  guarantees  fast  and  effective 
tracking  of  a sought  resource  and  enables  greater  production  visibility.  In  the  case  of  the  presented  repository, 
cataloging  the  resources  generated  by  the  students  is  done  through  a specific  module  and  uses  its  own 
metadata  format,  given  that  Moodle  does  not  implement  its  own  metadata  format. 

The  information  for  each  course,  resource  and  assignments  is  stored  by  Moodle  in  the  database  and  can 
be  used  for  generating  metadata  in  any  established  standard,  in  our  case  Dublin  Core,  the  standard  used  by 
DSpace.  Aftterwards,  depending  on  the  metadata  defined  for  sending  to  the  repository,  the  information  is 
retrieved  and  is  sent  together  with  the  resource  to  DSpace.  In  the  case  of  the  works  by  the  students,  the 
context  information  of  the  Task  is  taken  as  metadata,  including  course  name,  year,  teacher  in  charge,  teacher 
email  address  and  data  about  the  Task  itself  such  as  author  or  authors,  email  addresses,  grade,  comments  by 
the  teachers  and  assignment.  Implementing  text  mining  can  result  in  the  retrieval  of  specific  information  and 
key  words  that  allow  for  a more  detailed  classification  of  the  material  to  aid  future  queries.  Fig.  4 shows  the 
cataloging  process  and  its  actors. 


Student 

Teacher 

\ 

2/ 

1 \ 

/ 

/ 

/ 

/ " 

Assignment 

GH  ^ 

<xml  /> 


Librarians 


4 


Figure  4.  Resource  cataloging  shared  among  authors  (students),  revisers  (teachers)  and  additional  revisers  (librarians) 
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It  is  important  to  emphasize  that  while  most  of  the  data  is  extracted  automatically,  the  teacher  and  the 
student  can  incorporate  additional  information  such  as  additional  authors,  platform  and  programming 
languages  used  for  coding  the  resource. 

The  repository  can  also  be  accessed  through  Meran,  a free  software  system  developed  by  the  UNLP  for 
full  library  user  and  resource  management.  Librarians  can  also  catalog  these  resources  as  part  of  a certain 
specialty,  by  means  of  virtual  shelves  associated  with  a course,  numbering,  among  others. 

4.2  The  SWORD  Protocol 

The  SWORD  protocol  (Simple  Web-service  Offering  Repository  Deposit)  is  an  APP  (Atom  Publishing 
Protocol)  application  that  defines  simple  remote  content  storage  into  a repository  from  other  applications. 

With  only  two  basic  operations,  SWORD  creates  services  that  offer  functionalities  such  as  deposit  from 
multiple  locations  or  from  standard  applications,  and  multiple  storage  to  different  repositories.  The 
availability  of  SWORD  libraries  in  multiple  languages,  such  as  PHP,  Java,  Python  and  Ruby  promotes  the 
use  of  this  kind  of  integration.  DSpace,  Fedora  Eprints,  IntraLibrary,  and  DataBank  are  examples  of 
repositories  that  implement  this  service. 

An  interesting  aspect  of  this  protocol  is  that  the  sending  service  can  be  configured  so  that  any  user  can 
upload  information  directly  to  the  repository  without  logging  in.  This  solves  the  problem  of  worrying  about 
the  amount  of  users  or  permissions,  as  this  process  would  be  managed  through  the  application,  and  later,  a 
qualified  repository  user  could  validate  and  complement  the  received  data. 

4.3  Implementing  Communication 

After  analyzing  the  architecture  of  DSpace  it  became  apparent  that  no  implementation  or  installation  was 
necessary,  as  the  repository  incorporated  the  communication  functionality  through  this  protocol.  In  version 
1.8,  DSpace  implemented  the  protocol  in  two  ways,  as  a server  and  as  a client,  both  compatible  with 
SWORD  v2. 

In  contrast,  Moodle  does  not  offer  this  communication  functionality,  which  is  why  it  was  necessary  to  use 
the  Client  API  provided  by  SWORD  to  implement  a specific  module  that  will  retrieve  the  requested 
information  and  prepare  it  according  to  the  standards  established  by  the  protocol,  to  send  it  to  the  repository. 

For  our  application  case,  the  implemented  module  must  retrieve  the  file  stored  in  the  Moodle  Assignment 
and  then  build  the  package  in  the  SWORD  format  with  the  corresponding  metadata.  Following,  we  describe 
the  process  of  package  creation,  metadata  assignment  and  package  sending  between  the  two  platforms. 

4.3.1  Creating  the  Package 

As  previously  mentioned,  Moodle  does  not  have  a metadata  standard,  it  stores  basic  information  in  its 
database  tables,  which  can  be  used  to  generate  metadata  in  an  established  standard  such  as  Dublin  Core  in 
our  case. 

Moodle  has  its  own  classes  to  handle  the  File  System  and  the  files  themselves.  When  the  requested  file  or 
files  are  retrieved  through  the  Stored  File  class,  resource  data  is  retrieved  including  name,  creation  date,  last 
date  of  modification,  author,  format,  and  type  of  resource  within  Moodle,  among  others.  Information  such  as 
Assignment  name,  course  name,  and  platform  name  can  be  extracted  from  resource  context  data. 

The  application  generates  a METS  (Metadata  Encoding  and  Transmission  Standard)  [10]  package  with 
files  and  administrative,  descriptive  and  structural  metadata  that  the  repository  will  use  to  incorporate  the 
resource.  The  metadata  are  placed  in  an  xml  file  that  includes:  a heading,  descriptive  metadata  in  MODS 
format.  Creative  Commons  license,  list  of  files  and  structural  map.  Later,  this  set  of  files  will  be  packed  in  a 
single  zip  file. 

4.3.2  Sending  the  Package 

The  package  is  sent  through  the  instantiation  of  a client  of  the  PHP  API  provided  by  SWORD.  The  function 
deposit  is  used  to  automatically  deliver  it  to  the  repository.  It  is  necessary  to  specify  in  this  function  the  URL 
of  the  repository,  an  access  account  and  password  if  necessary,  the  content  format,  the  destination  collection 
and  the  zip  file  with  the  content.  Pig.  5 shows  the  steps  followed  when  sending  a resource  from  Moodle  to 
DSpace. 
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Figure  5.  Communication  scheme  between  Moodle  and  DSpace  through  SWORD 
The  steps  are  as  follows: 

1 The  students  deliver  their  assignment  by  uploading  a file. 

2 The  teacher  follows  the  deliveries  and  the  corresponding  corrections,  sending  feedback  to  the 
students  when  necessary.  Once  the  delivery  is  finished,  the  teacher  can  choose  to  send  it  to  the 
repository. 

3 Once  the  teacher  decides  to  make  a delivery,  the  module  comes  into  play.  It  takes  each  one  of  the 
assignments  and  packs  them  with  their  corresponding  metadata  into  a zip  file. 

4 The  package  is  sent  through  SWORD  to  a specific  collection  in  the  chosen  repository. 

5 The  default  SWORD  module  in  DSpace  unpacks  it  and  retrieves  the  Dublin  Core  metadata  and  files. 

6 Each  of  the  files  is  added  as  items  to  the  specific  collection. 

7 DSpace’s  SWORD  module  sends  an  xml  response  to  the  Moodle  newly  created  module  indicating 
operation  status. 

8 Librarians  catalog  the  resources 

After  sending,  the  complete  resource  is  stored  in  the  corresponding  collection  within  the  repository,  with 
its  associated  metadata.  The  result  of  this  operation  is  the  same  as  if  the  item  was  added  manually  following 
the  steps  required  by  DSpace  for  resource  storage. 


5.  CONCLUSIONS 

In  e-learning,  communication  among  platforms  facilitates  exchange,  integration  and  reuse  of  educational 
resources  that  have  been  developed  in  heterogeneous  platforms  and  tools.  Standard  creation  and  adoption 
established  the  use  of  certain  norms  for  digital  content  creation  and  communication  among  systems.  The 
growth  of  distance  education  has  resulted  in  users  placing  emphasis  in  technical  aspects  related  to  connecting 
and  using  distributed  resources  in  other  platforms  such  as  digital  repositories. 

The  communication  implemented  between  the  Moodle  LMS  and  the  DSpace  repository  allows  for  a 
reduction  in  the  overload  of  storing  the  same  resources  in  multiple  platforms.  This  encourages  the  teachers 
that  manage  their  courses  in  the  educational  platform  to  publish  one  or  more  student  projects  delivered  in  one 
or  multiple  external  repositories,  as  they  do  not  need  to  be  familiar  with  their  interface  and  mode  of  access. 
Moreover,  the  module  implemented  prepares  the  information  to  be  transferred  by  incorporating  metadata 
with  course  context  information,  in  the  standard  format  used  by  destination  repositories.  Semi-automatic 
publishing  of  material  is  more  practical  as  it  is  not  necessary  to  follow  all  the  steps  required  within  the 
repository  to  upload  the  resource  and  complete  the  metadata  that  are  indispensable  for  search  and  retrieval.  In 
our  case,  the  module  is  being  tested  with  the  DSpace  repository,  and  the  transferred  resource  remains  with  its 
basic  Dublin  Core  metadata  complete.  The  functionality  of  this  module  can  be  extended  to  other  repositories. 
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adapting  the  way  in  which  material  is  organized,  taking  into  account  the  standards  used  in  each  specific 
platform.  To  incorporate  specific  metadata  that  match  other  criteria  and  goals  classification  is  planned  the 
intervention  of  an  interdisciplinary  group  of  librarians  who  contribute  their  expertise  on  cataloging.  In  our 
project,  librarians  participation  is  done  at  a later  stage  in  the  teacher  publishing  process.  This  will  complete 
the  registration  of  metadata  with  resource  specific  information  from  external  aspects  and  offer  different 
contribution  when  performing  searches. 
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ABSTRACT 

Libraries  are  important  to  universities,  and  they  have  two  primary  features:  readers  as  well  as  collections  are  highly 
professional.  In  this  study,  based  on  the  experimental  study  with  five  millions  of  users’  borrowing  records,  our  discussion 
covers:  (1)  the  necessity  of  recommender  system  for  university  libraries;  (2)  collaborative  filtering  (CF)  technique  is 
applicable  and  feasible;  (3)  user-based  CF  technique  is  preferred  over  item-based;  (4)  the  performance  of  applying 
classical  used-based  collaborative  filtering  algorithm;  (5)  the  effectiveness  of  local  recommendation  and  the  great  saving 
of  computing  resource  it  may  bring  potentially.  Since  the  data  size  used  in  our  experiments  is  the  largest  one  among 
similar  studies,  it  is  believed  a valuable  reference  on  this  specific  direction. 


KEYWORDS 

Recommender  system,  university  library,  collaborative  filtering,  user  modeling 


1.  INTRODUCTION 

In  most  universities,  libraries  are  well  equipped  with  modern  IT  infrastructure,  and  large  amount  of  usage 
data  have  chance  to  been  collected  and  stored.  As  compared  with  public  libraries,  university  libraries  have 
two  primary  features,  (1)  large  volume  of  abstruse  collection  on  science  and  engineering,  and  (2)  readers 
have  in-depth  knowledge  on  related  fields.  Meanwhile,  due  to  the  explosion  of  knowledge,  the  scale  of 
collection  increases  quickly  as  well. 

For  a long  time,  we  depend  on  the  search  engine  to  retrieve  items  (or  books)  in  the  collection.  To  do  so, 
we  may  have  to  iteratively  try  different  keywords,  and  adjust  them  given  the  results  if  necessary.  It  repeats 
until  satisfactory  items  are  found.  During  this  procedure,  those  who  know  how  to  represent  their  requests 
clearly,  especially  in  a manner  “liked”  by  the  engine,  may  reach  their  goals  soon.  Besides,  users’  carefully 
made  decision  and  options  could  not  benefit  others  who  have  similar  requests,  although  the  system  has 
helped  to  record  who  has  borrowed  which  books  at  what  time.  Hence,  in  conclusion,  search  engine  is  not 
enough  for  us  to  effectively  find  what  we  want  in  university  library,  and  we  desire  a smarter  assistant  which 
could  make  use  of  peers’  options.  In  this  project,  we  work  to  build  a suitable  recommender  for  university 
library,  but,  in  this  submission,  we  only  share  what  we  have  gained  from  some  early  experiments.  The 
discovery  may  guide  our  future  work. 

This  paper  is  structured  as  follows.  In  Section  2,  related  works  are  reviewed  and  our  contributions  are 
listed.  In  Section  3,  we  analysis  the  necessity,  feasibility  and  a potential  solution  based  on  the  specific 
requirements  of  university  libraries.  Then,  in  Section  4,  we  introduce  two  primary  collaborative  filtering 
algorithms,  and  discuss  the  cause  of  our  selection.  In  Section  5,  a series  of  experiments  are  conducted,  using 
about  five  millions  of  records  from  one  Chinese  university,  Huaqiao  University.  We  conclude  in  Section  6. 
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2.  RELATED  WORK  AND  OUR  CONTRIBUTION 

There  are  some  published  works  on  the  recommender  system  for  university  library.  According  to  whether 
there  are  reported  experiments  with  real  data  as  contained,  we  filtered  out  with  only  trustable  ones  left, 
including  contributions  based  on  clustering  [1,  2],  association  analysis  [3,  4]  and  collaborative  filtering  (CF) 
[5,  6]  respectively.  Though  our  discussion  focuses  on  CF  as  well,  it  has  some  obvious  differences: 

1.  We  analysis  the  necessity  in  a quantitative  way  (see  next  section),  while  [1-6]  only  provide 
qualitative  analysis; 

2.  Like  [1,  3,  4],  we  directly  use  the  original  borrowing  records  like  <user_id,  book_id,  timestamp>,  i.e. 
each  one  indicates  someone  borrowed  some  book  at  some  time.  However,  in  [5],  it  uses  the 
circulation  log  of  the  reading  room.  In  [6],  since  it  provides  online  reading,  a score  is  estimated  as  the 
preference  on  behalf  of  the  user,  based  on  how  many  pages  viewed.  Borrowing  log  is  used  in  [1]  as 
well,  but  a score  is  also  calculated  according  to  the  borrowing  length  (i.e.  how  long  the  reader  keep 
the  book)  and  whether  renewal  happens; 

3.  For  the  first  time,  we  discovered  that  local  search  is  effective  to  construct  top-N  recommendation  list 
while  applying  CF-based  recommender  for  university  library  (see  Section  5).  This  would  bring  great 
reduction  on  computing  complexity  as  compared  with  conventional  global  search  way; 

4.  Previous  studies  only  used  tiny  or  small  scale  of  usage  log  in  their  experiments.  For  example,  only 
38,078  records  were  used  in  [5];  data  about  2,358  users  and  4,352  books  were  used  in  [4];  7,090  loan 
records  between  January  and  March,  2003  in  [3];  records  collected  between  September  2006  and 
May  2007  were  used  in  [1];  two  semesters  of  records  in  [2].  In  our  study,  totally  4,932,579 
borrowing  records  are  used,  with  a span  of  10  years,  and  it  is  at  least  ten  times  of  the  largest  one 
found  in  previous  studies.  Hence,  it  is  closer  to  the  real  application  scenario. 


3.  NECESSITY  AND  EEASIBILITY 

How  to  bridge  the  ‘gap’  between  the  professional  readers  and  involved  scholarly  collections  is  one  of  the 

most  challenging  tasks  faced  by  university  libraries.  Recommender  system,  due  for  its  success  in  many  fields 

in  the  past  two  decades,  is  believed  an  economic  and  workable  solution. 

3.1  Necessity 

The  necessity  of  building  recommender  system  for  university  library  can  be  explained  with,  but  not  limited 

to,  the  following  points: 

> Fundamental  information  system  in  library  is  mature,  including  easy-to-use  text  retrieval  function. 
However,  we  have  to  transform  our  own  abstract  and/or  abstruse  requests  into  one  or  few  keywords, 
facing  the  traditional  search  engine.  This  step  demands  intensive  brain  work,  and  it  may  greatly 
influence  the  quality  of  retrieval.  Worse  thing  is  that  it  only  returns  text-related  results,  preventing  us 
from  accessing  books  unknown  but  may  interested  us  [7] ; 

> Meta  information  of  each  book  in  the  database  is  rare,  which  furthermore  restricts  retrieval  function.  For 
example,  most  books  in  our  university’s  collection  only  contain  basic  sections  like  titles,  authors  and 
publishers,  but  no  abstract,  saying  nothing  of  contents.  The  shortage  of  these  rich  information  will  make 
current  text  retrieval  function  work  poorly; 

> Low  penetration  rate  due  to  the  fact  that  our  collection  is  abstruse.  For  example,  in  our  library,  totally 
there  are  around  two  millions  of  volume  and  450  thousands  unique  items.  However,  only  47.51%  of 
(unique)  items  once  be  borrowed.  Can  we  declare  that  those  remaining  items  are  just  not  interesting?  Or 
can  we  say  that  they  are  not  known  by  readers?  I think  the  second  is  more  likely,  so  we  need  a ‘smart’ 
recommender  system  to  help  readers  in  university  library  find  what  they  want,  especially  those  they 
don’t  know  yet  but  may  feel  interested; 

> We  hope  such  a system  won’t  disturb  us,  e.g.  we  need  to  inform  it  what  we  want  now  and  then 
explicitly,  but  just  use  the  recorded  our  usage  data  and  work  ‘silently’.  By  assuming  that  we  borrow  a 
book  only  when  we  like  it,  existing  recommender  technique  is  an  ideal  candidate  since  it  could  use  these 
implicit  data  to  produce  something  useful. 
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3.2  Feasibility 

There  are  two  primary  information  filtering  techniques;  content-based  [8,  9]  and  collaborative  filtering-based 
[10,  11].  Most  methods  used  in  content-based  come  from  information  retrieval,  and  it  has  limits  in,  at  least, 
three  aspects; 

> It  works  by  comparing  if  content  features  are  matched  with  user  profiles.  To  do  so,  we  need  some  way 
to  extract  features  of  the  target  content,  and  furthermore,  we  want  the  features  to  be  complete  as  well  as 
concise.  However,  there  is  no  effective  approach  to  do  feature  selection,  especially  when  we  face 
multimedia  resource,  and  extracted  features  alone  cannot  tell  us  the  quality  of  information,  which  may 
influence  users’  satisfaction  level.  Finally,  our  fact  is  that  we  only  have  rare  information  about  the  book, 
as  discussed  above; 

> Even  though  we  have  enough  information  on  content,  constructing  users’  profile  is  again  a challenging 
task  since  it  may  also  rely  on  the  feature  extractions  of  contents; 

> Content-based  filtering  technique  may  disturb  users  since  their  privacy  would  be  ‘touched’,  due  to  the 
inborn  mechanism  of  this  approach. 

Hence,  collaborative  filtering  (CF)  based  recommendation  is  the  remaining  option.  The  most  feature  of 
this  technique  is  its  independence  of  content,  therefore  it  is  applicable  to  not  only  text  but  multimedia.  Its 
basic  assumption  is  that  users  similar  to  each  other  may  have  similar  interest,  i.e.  borrowing  similar  books 
here.  Some  observations  and  features  of  this  application  are  listed  as  below,  and  they  are  solvable  by  CF 
technique,  or  useful  to  the  success  of  this  technique; 

> Meta  data  about  books  is  rare.  Besides,  on  most  conventional  library  system,  readers  have  no  way  to 
comment  or  give  explicit  feedback  online,  though  this  is  quite  common  on  Internet  today; 

> Readers’  borrowing  records  can  be  viewed  as  their  profile.  Since  readers  normally  make  the  selection 
only  after  careful  checking  and  consideration,  circulation  record  could  be  a strong  indication  that 
whether  one  user  ‘like’  a book; 

> Enough  readers’  borrowing  records  are  collected.  Totally  we  have  around  five  millions  of  borrowing 
records.  Because  of  the  relatively  high  frequency  of  circulation  in  university,  new  records  emerge  and 
are  collected  every  day,  which  could  be  used  to  update  users’  profile; 

> Borrowing  behavior  is  of  implicit  type  of  feedback,  without  having  to  modify  the  existing  system,  and 
bring  no  interrupt  during  users’  interaction  with  system; 

> Implementation  based  on  CF  technique  may  minimize  the  influence  on  existing  three-tier  system 
(Web/Application/Database).  Because  what  data  required  by  the  recommender  can  be  retrieved  from 
database  directly,  we  need  not  intervene  with  application  layer.  For  security  and  performance 
consideration,  we  may  create  an  independent  database  on  the  recommender  side,  and  synchronize  it 
with  online  database  according  to  acceptable  strategy,  e.g.  incremental  updating  during  low  transaction 
volume  period; 

> Outputs  by  the  recommender  can  be  exposed  with  REST  (Representational  State  Transfer)  interface, 
and  easily  be  plugged  into  existing  application  systems  (see  Figure  1).  For  example,  what  produced  by 
our  recommender  may  be  presented  as  “Users  who  borrow  this  book  also  borrow  them”; 


Figure  1.  Overall  logical  view  of  the  existing  application  and  the  recommender 
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> Research  on  collaborative  filtering  has  been  lasted  for  over  15  years,  and  there  are  many  successful 
applications  [12-14]. 

Therefore,  CF  is  chosen  in  this  project.  Like  clustering  algorithms,  CF  also  depends  on  the  search  of  near 
neighbors,  but  CF  can  provide  more  fine-grained  results.  Besides,  classical  k-means  algorithm  also  requires 
users  to  specify  the  choice  of  parameter  k.  As  compared  with  association  analysis,  CF  could  tell  us  more 
previously  unknown  knowledge  from  similar  users,  and  in  a faster  way.  To  avoid  generating  too  many  rules, 
association  analysis  has  to  increase  the  support  and  confidence  level,  which  will  influence  the  discovery  of 
some  new  items. 


4.  COMPARISON  OF  CF  ALGORITHMS 


4.1  Overview 


There  are  two  kinds  of  CF  algorithms,  user-based  [10]  and  item-based  [11].  Both  use  user-item  score  matrix 
and  some  active  user  as  input,  and  produce  an  item  recommendation  list  for  this  active  user. 

The  so-called  user-item  score  matrix  is  the  core  data  structure  in  CF  algorithm,  and  it  is  of  size  m X n,  i.e. 
m users’  preferences/options  on  n items.  It  is  denoted  as  S^xn-  The  value  of  each  cell  5(u,  t)  expresses  user 
’s  preference  on  item  t,  and  it  can  either  be  boolean  or  numeric.  In  our  application,  S(u,  t)  is  boolean,  with  1 
indicates  that  user  u once  borrowed  item  i and  0 for  not.  Similarly,  ^(u,-)  refers  to  user  m’s  circulation  vector, 
and  S(-,y)^for  the  circulation  vector  about  item  j. 

Given  S^xn  > the  kernel  step  in  CF  is  to  determine  the  similarity  between  users/items.  Widely  used 
measures  include  cosine  similarity,  Pearson’s  correlation,  various  revised  cosine  similarity  and  Tanimoto 
coefficient  (also  called  Jaccard  index/coefficient  often)  for  binary  data  [15],  which  can  easily  be  found  in 
nearly  all  statistics  books.  In  our  application,  Tanimoto  coefficient  is  used,  and  its  equation  in  user-based 
scenario  is  shown  below: 


sim{u,v') 


■ 
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The  corresponding  equation  given  item-based  calculation  is  similar 
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4.2  User-based  CF 

It  is  the  earliest  CF  algorithm  being  proposed.  Based  on  the  calculation  of  user  similar  (see  equation  (1)) , we 
are  able  to  find  k users  closest  to  the  active  user  a.  Then,  for  each  item  x E [Ju=iij\^(dt>D  — 1}\ 
{yl^Ca,)/)  = 1}  , we  can  predict  user  a’s  possible  preference  on  x as  below: 

S(a,x)  = 

By  sorting  all  such  S (a,  x)  in  descending  order,  we  could  produce  a recommendation  list  containing  N 
items  with  the  highest  score,  i.e.  the  known  top-  N recommendation. 

4.3  Item-based  CF 

For  some  online  recommender,  the  Tong  tail’  effect  of  users  is  more  prominent  than  item’s.  For  example, 
there  are  hundreds  of  millions  of  users,  but  millions  of  items,  on  Amazon.com.  Furthermore,  their  users’ 
amount  and  profiles  changes  all  alone,  so  we  have  to  calculate  the  similarity  of  users  online  if  we  hope  to 
catch  users’  updates  in  time,  which  will  slow  down  the  response  speed  of  recommender  built  upon  user-based 
CF  technique.  Differently,  similarities  between  items  are  relatively  stable,  so  their  similarity  could  be 
calculated  offline  in  item-based  CF,  and  be  updated  periodically  (and  also  offline),  which  ensures  the  speed 
of  online  response.  Hence,  item-based  CF  is  preferred  on  most  large  scale  Web  sites  when  construct  their 
own  recommender  systems  [13,  16]. 
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4.4  Our  Choice 

In  our  project,  used-based  CF  algorithm  is  finally  selected  based  on  the  following  causes: 

> There  are  around  70  thousands  of  users  and  500  thousands  of  books  (items)  in  database,  and  more  items 
are  added  to  the  collection  than  newly  enrolled  students  per  year; 

> Item-based  CF  will  compute  the  prediction  on  an  item  i for  a user  u by  computing  the  sum  of  the  ratings 
given  by  the  user  on  the  item  similar  to  i,  and  each  ratings  is  weighted  by  the  corresponding  similarity 
sim(i,j)  between  items  i and  j.  However,  the  ‘interest’  of  readers  in  university  is  NOT  stable  - it  may  be 
completely  different  between  adjacent  semesters.  Hence,  what  one  read  in  this  semester  may  have  little 
influence  on  what  s/he  would  select  in  the  next  semester,  which  limits  the  ‘power’  of  item-based  CF 
algorithm; 

> Based  on  our  experimental  results,  readers’  interest  is  rather  stable  (see  the  next  section)  in  macroscopic 
perspective,  i.e.  limited  to  their  profession.  This  means  that  readers  from  the  same  major  will  have  share 
on  reading  selections,  and  this  is  the  basis  of  user-based  CF. 


5.  EXPERIMENTS:  DESIGN,  DISCOVERY  AND  DISCUSSION 

There  are  two  kinds  of  evaluation  for  recommender  systems,  online  and  offline.  Online  evaluation  requires 
the  recommender  to  work  online  with  existing  application  to  collect  actual  interaction  log.  Its  cost  is  high, 
and  it  is  not  suitable  for  repeating  test,  especially  when  the  work  is  not  mature  since  it  may  sacrifice  the 
experience  of  users.  Most  research  work  on  recommender  takes  offline  experiments,  i.e.  feed  recommender 
system  with  partial  history  data  to  train  it,  and  compare  the  predicated  value  with  those  parts  left  but 
observed  in  history  log. 

5.1  Dataset 

Dataset  used  in  this  project  is  from  the  library  of  Huaqiao  University,  ranging  from  1998  to  2011.  There  is 
various  information  as  contained,  but  we  only  use  the  users’  borrowing  records,  basic  profile  (including 
which  department  s/he  belongs  to)  and  brief  description  of  books.  Related  summary  statistics  can  be  found  in 
Table  1. 


Table  1.  Summary  statistics  of  dataset 


Total  # of  borrowing  records 

4,932,579 

Total  # of  books  in  system 

445,722 

Total  # of  books  with  borrowing  record 

211,776 

Total  # of  users  in  system 

76,524 

Total  # of  users  with  borrowing  records 

65,483 

Avg.  # records  per  users 

75 

Avg.  # records  per  books 

23 

Regarding  books,  we  only  count  the  number  of  dijferent  books,  ignore  multiple  copies 

In  the  current  dataset,  due  to  some  unknown  reason  (probably  early  users’  borrowing  records  were  not 
saved),  there  are  1 1,041  users  with  no  borrowing  records.  Even  we  remove  these  users  and  those  books  with 
no  records  from  consideration,  the  user-item  matrix  is  still  very  sparse,  about  0.036%  (=4,932,579/(65,483 
211,776)). 

We  also  extract  summary  statistics  about  two  typical  colleges.  Materials  Science  and  Engineering  (MSE) 
and  Public  Administration  and  Service  (PAS),  as  Table  2. 


Table  2.  Summary  statistics  of  two  colleges:  MSE  and  PAS 


College  of  MSE 

College  of  PAS 

# of  lending  records 

548,998 

270,351 

# of  users  with  records 

7,277 

3,377 

# of  books  covered 

87,745 

63,925 

Avg.#  of  records  per  usei 

75 

80 
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5.2  Experimental  Design  and  Discussion 

It  is  mentioned  (in  last  section)  readers  from  the  same  college/department  would  be  similar  one  another  on 
reading  interest,  so  we  could  only  refer  to  these  users’  preferences  during  the  decision  making  in  used-based 
CF.  If  this  could  be  confirmed,  we  need  only  conduct  a local  search  of  neighbors,  instead  of  looking  for 
similar  users  from  the  whole  user  set.  To  do  so,  we  select  MSB  and  PAS  colleges  as  examples.  We  compare 
the  recommendation  effect  on  MSB  and  PAS  given  a recommender  trained  by  the  whole  dataset  (see  Table  1) 
and  dataset  related  to  these  two  colleges(see  Table  2)  respectively. 

Given  the  dataset,  we  take  the  10-folder  strategy,  i.e.  dividing  the  data  into  ten  groups,  selecting  one  for 
testing  and  the  remaining  ones  for  training.  By  repeating  the  experiments  for  ten  times,  we  measure  the 
average  precision  and  recall.  Although  there  are  various  measures  proposed  to  evaluate  one  recommender 
[17],  such  as  coverage  and  serendipity,  accuracy  is  still  the  most  concerned  index. 

As  mentioned  in  last  section,  we  implement  the  recommender  as  user-based,  and  Tanimoto  coefficient  is 
applied  to  measure  the  similarities.  We  select  K neighbors,  and  study  the  top-A  recommendation  list.  By 
differentiating  K and  N,  we  have  the  chance  to  compare  their  influence  on  recommendation  accuracy. 

5.3  Recommender  trained  with  whole  Dataset 


Birstly,  we  train  the  recommender  with  whole  dataset,  and  use  it  to  provide  recommendation  list  for  MSB  and 
PAS  colleges.  This  is  the  typical  mechanism  in  conventional  CB-based  recommender.  In  experiments,  K is 
set  as  10,  20,  30,  40  and  50  respectively;  N is  set  10,  20  and  30. 

It  is  observed  from  Bigure  1 that,  (1)  Both  precision  and  recall  increase  with  larger  K,  e.g.  the  precision  of 
MSB  has  34%  increase  when  K increases  from  10  to  50  given  iV  = 10;  (2)  Given  the  same  K , smaller  iV 
results  with  higher  precision  but  lower  recall.  Bor  example,  the  precision  of  PAS  when  N =10  is  about  38% 
higher  when  N =30  given  K=10\  while  the  corresponding  recall  when  N =10  is  only  46%  of  that  when  N 
=30. 

MES  PAS 


Recall 


Precision 


Figure  2.  Average  precision  and  recall  about  MSB  and  PAS  by  recommender  trained  with  the  whole  dataset 

5.4  Recommender  trained  with  Partial  Dataset 


Secondly,  we  train  the  recommender  with  partial  dataset.  Bor  instance,  a recommender  is  trained  by  the  MSB 
dataset,  and  used  to  provide  recommendation  list  for  readers  belonging  to  MSB.  This  is  different  from  the 
conventional  solution,  and  the  remaining  settings  are  kept  the  same. 

MES  PAS 


Precision 


Figure  3.  Precision  and  recall  about  MSE  and  PAS  by  recommender  trained  with  only  MSE  and  PAS  dataset  respectively 
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Similar  trend  as  in  Figure  2 is  observed  from  Figure  3,  (1)  both  precision  and  recall  increase  with  larger 
K\  (2)  given  same  K , smaller  N results  with  higher  precision  but  lower  recall. 

5.5  Global  vs.  Local  Recommendation 

For  easy  reference,  we  call  recommender  trained  by  whole  dataset  as  global  recommendation,  and  that  by 
partial  dataset  as  local  recommendation. 

Table  3.  Precision  (PI)  and  Recall(Rl)  achieved  by  whole  data  VS.  Precision(P2)  and  Recall  (R2)  achieved  by  partial 

data  (P2)  about  MSB’s  users:  (P1/P2)  and  (R1/R2) 


N 

K \ 

10 

20 

30 

P1/P2 

R1/R2 

P1/P2 

R1/R2 

P1/P2 

R1/R2 

10 

1.03 

1.03 

1.02 

1.02 

1.01 

1.00 

20 

1.01 

1.01 

0.99 

0.99 

1.02 

1.01 

30 

1.01 

1.01 

0.98 

0.98 

0.98 

0.98 

40 

0.99 

0.99 

0.98 

0.98 

0.98 

0.97 

50 

0.99 

0.99 

0.98 

0.98 

0.98 

0.98 

Table  4.  Precision  (PI)  and  Recall(Rl)  achieved  by  whole  data  VS.  Precision(P2)  and  Recall  (R2)  achieved  by  partial 

data  (P2)  about  PAS’s  users:  (P1/P2)  and  (R1/R2) 


N 

K \ 

10 

20 

30 

P1/P2 

R1/R2 

P1/P2 

R1/R2 

P1/P2 

R1/R2 

10 

1.16 

1.12 

1.16 

1.13 

1.16 

1.12 

20 

1.11 

1.07 

1.09 

1.06 

1.11 

1.08 

30 

1.06 

1.03 

1.09 

1.06 

1.10 

1.07 

40 

1.06 

1.03 

1.12 

1.08 

1.12 

1.09 

50 

1.10 

1.06 

1.10 

1.07 

1.12 

1.08 

Table  3 and  Table  4 are  about  the  comparison  of  precision  and  recall  achieved  by  global  and  local 

recommendation  on  MSB’s  and  PAS’s  readers  respectively.  It  can  be  observed  and  concluded  that: 

> Local  recommendation  may  even  achieve  higher  precision  and  recall  than  global  recommendation, 
which  can  be  observed  from  the  gray  cells  in  Table  3.  However,  this  is  NOT  observed  in  Table  4,  which 
may  be  explained  that  engineering  readers’  (e.g.  college  of  MSB  here)  interest  is  NOT  as  diverse  as  that 
of  non-engineering  readers  (e.g.  college  of  PAS  here); 

> Regarding  MSB’s  readers,  by  average,  the  global  recommendation  achieves  10.9%  and  7.6%  higher 
precision  and  recall  respectively  than  local  recommendation.  Considering  the  great  computing 
complexity  saved  by  local  recommendation,  this  cost  is  affordable; 

> K=30  and  N=20  are  the  suggested  parameters  observed  in  our  experiments. 


6.  CONCLUSION  AND  FUTURE  WORK 

In  this  article,  we  discuss  the  necessity  of  building  recommender  system  for  university  library  firstly.  Then, 
we  compare  and  propose  that  user-based  CB  algorithm  is  more  suitable  for  this  application.  We  conduct 
experiments  with  five  millions  of  lending  records  collected  in  one  Chinese  university.  National  Huaqiao 
University.  The  results  confirm  that  we  could  train  individual  recommender  for  each  college  (or  even 
department),  or  at  least,  show  recommendations  from  these  individual  recommenders  with  higher  priority 
than  those  from  global  recommender.  This  finding  could  bring  great  reduction  on  computing  complexity, 
with  affordable  cost  on  recommendation  accuracy. 

Although  we  focus  only  on  accuracy  here,  we  are  interested  to  study  the  serendipity  in  future  study, 
aiming  at  better  service.  However,  till  now,  there  is  no  mature  definition  and  measure  on  serendipity. 
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Meanwhile,  how  to  tradeoff  the  accuracy  and  serendipity  is  an  interesting  topic  as  well.  More  experiments 
will  be  designed  and  conducted  to  help  us  gain  more  knowledge  and  experience  about  the  application  of 
recommender  in  university  library,  before  we  start  to  develop  the  real  system. 
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ABSTRACT 

The  political  will  to  move  educational  opportunities  online  is  growing  for  numerous  reasons  and  new  mobile 
technologies  are  being  adopted  at  unprecedented  rates.  Such  a context  presents  opportunities  to  develop  online  programs 
and  experiments  in  universities,  with  new  affordances  to  solve  old  problems  such  as  access  and  isolation. 

This  paper  presents  an  initial  experiment  in  the  creation,  development  and  first  implementation  of  an  online  Bachelor’s 
program  in  adult  education  and  digital  technology.  It  was  designed  with  a central  videoconferencing  component  intended 
to  reduce  isolation  for  the  students  and  technical  affordances  were  selected  to  promote  course  development  for  both  a 
mobile  implementation  and  for  at  least  partial  open  access. 

The  results  of  this  initiative  have  both  demonstrated  the  potential  of  such  an  approach  as  well  as  allowed  some  of  the 
finer  elements  to  emerge,  such  as  expectations  on  all  parts  and  administrative  adaptation  to  this  new  reality. 


KEYWORDS 

Videoconferencing,  social  networking,  m-Learning,  online  program,  curriculum  development 


1.  THE  CONTEXT 

People  are  social  beings  (Lowenthal,  2010).  They  rely  on  all  senses  for  connecting  with  others 
(Gunawardena,  1995),  and  they  are  constantly  on  the  move.  This  implies  that  for  any  education  endeavour  to 
be  successful,  it  must  not  only  allow  for,  but  promote  learning  in  groups  or  teams  or  communities.  The 
learners  much  prefer  to  be  able  to  interact  with  others  in  a direct  a manner.  Consequently,  the  live,  face-to- 
face  discussion  is  generally  preferred  over  all  other  modes  of  interaction.  The  learning  situations  must  also 
address  all  senses  and  all  dimension  of  the  human  experience  for  it  to  be  relevant.  The  written  word  is  fine 
for  getting  to  information,  but  to  access  rich  thick  information,  the  experience  of  involving  sights,  sounds, 
and  even  interacting  with  objects,  fosters  the  construction  of  multidimensional  knowledge.  Finally,  in  this 
age  of  short  careers  and  constant  need  for  re-education  and  lifelong  learning,  this  upgrading  of  one’s 
education  is  done  on  the  run,  in  transit,  as  less  time  is  available  to  commit  long  periods  in  a day  to  one  single 
activity. 

These  three  elements  of  human  nature  present  challenges  for  post  secondary  education,  particularly 
university  education,  as  it  is  facing  yet  another  milestone  in  its  continued  evolution.  With  a continued 
pressure  for  democratization  of  education  in  all  comers  of  the  world  reaching  the  tertiary  sector,  the  demands 
for  access  to  student  spaces  are,  by  far,  exceeding  the  capacity  of  a system  that  was  designed  for  the 
exceptions  rather  than  the  masses  (Trow,  2006).  As  the  higher  education  sector  has  been  scrambling  to  adapt 
to  this  new  reality  following  a recent  economic  downturn,  some  governments,  such  as  is  the  case  in  Ontario 
(Canada),  have  been  actively  initiating  conversations  to  find  new,  effective  and  efficient  ways  to  meet  this 
challenge.  Locally,  in  Ontario,  the  MTCU  (Ministry  of  Training  Colleges  and  Universities,  2012)  has 
recently  been  advocating  for  both  colleges  and  universities  to  increase  their  online  offerings  in  terms  of 
courses  and  programs.  This  political  will  provides  both  the  challenges  as  well  as  opportunities  to  support 
university  initiatives  to  explore  new  avenues  in  the  development  of  online  education. 

The  current  state  of  technological  development  as  well  as  its  extensive  adoption  by  populations  around 
the  globe  is  fostering  many  initiatives  in  online  education  aiming  at  addressing  these  challenges.  The  rapidly 
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increasing  use  of  synchronous  modes  of  communication,  facilitated  by  the  adoption  of  videoconferencing 
platforms  and  recently  adding  the  possibilities  of  group  video  meetings,  live  collaboration  at  a distance  is 
easy,  affordable  and  directly  fosters  a greater  sense  of  community,  even  online  (Mclnnerney,  & Roberts, 
2004).  In  addition,  these  live  media,  also  call  upon  much  more  than  textual  information.  They  make  use  of 
sight  and  sound.  They  promote  the  live  social  interaction  between  people  in  a more  complete  experience. 
Adding  to  this  the  fact  that  adult  learners  tend  to  select  their  programs  of  study  based  on  specialisation  rather 
than  the  proximity  of  the  institution,  the  availability  of  online  options  become  an  important  solution  for  both 
educational  institutions  and  the  learners.  Furthermore,  with  career  changes  becoming  far  more  frequent,  the 
need  for  constant  upgrading  of  skills  and  competencies  is  matched  only  by  the  increase  need  for  commuting 
to  the  next  job  or  project.  These  kinds  of  changes  tend  to  involve  more  time  spent  in  transit  or  in  remote 
locations  (hotels,  offsite  work  locations,  etc.)  and  the  possibility  of  dedicating  some  of  that  time  for  study  is 
beginning  to  surface  as  a desired  option.  Making  use  of  synchronous  communications  on  mobile  technology 
in  order  to  access  and  participate  in  formal  education  from  anywhere,  anytime  will  shortly  become  an 
absolute  necessity  for  a very  large  number  of  post-secondary  learners. 

In  the  midst  of  a multiplicity  of  online  education  initiatives  by  institutions  around  the  world,  members  of 
the  Faculty  of  Education  at  the  University  of  Ontario  Institute  of  Technology  (UOIT)  were  interested  in 
providing  an  alternative  form  of  program  which  would  capitalize  on  available  online  technology,  while 
paying  attention  to  the  learner  needs  for  socialisation,  multisensory  experiences,  and  mobility.  In  this  context, 
a project  was  initiated  to  create  a Bachelor  level  program  (B.A.)  that  would,  on  the  one  hand,  expand  on  the 
strengths  of  existing  programs  in  education,  and  on  the  other,  provide  the  impetus  for  developing  the 
necessary  experience  in  online  learning  to  eventually  integrate  mobile  learning  as  a regular  affordance  in  all 
online  courses.  Such  a program  would  have  to  be  designed  in  a manner  as  to  have  the  modes  of  interaction 
between  all  participants  (professors  and  students),  as  well  as  being  fully  consistent  with  the  objectives  and 
content  of  all  the  courses.  As  initial  internal  faculty  discussions  started,  the  idea  of  using  desktop  (or  mobile) 
videoconferencing  (VC)  and  social  networking  quickly  emerged  both  as  the  target  and  the  necessary  context. 

It  was  assumed  that  the  VC  proposition  would  attract  some  resistance  and  consequently  attention  was 
paid  to  attempt  to  understand  and  address  the  major  concerns.  The  most  important  criticism  was  that  online 
learning  was  perceived  as  cold  and  isolating,  one  where  the  student  interacts  mostly  with  online  documents 
as  well  as  with  sparse  and  impersonal  communications  through  e-mail  or  disembodied  discussion  forums 
(Goldrick,  O’Higgins  Norman,  2012).  Recognising  that  creating  a meaningful  learning  experience  requires 
the  establishment  of  a community  of  personal  relationships  between  the  students  themselves  as  well  as  with 
their  professors  and  tutors,  the  concept  of  using  more  face-to-face  communications  became  a central 
objective  for  the  new  program.  Therefore,  it  was  proposed  that  all  courses  should  include  regular  (weekly) 
meetings  using  some  form  of  VC  allowing  students  to  access  from  any  location  that  was  equipped  with 
broadband  availability.  It  was  also  suggested  that  these  meetings  would  serve  to  foster  rich,  meaningful 
discussions  and  interactions  that  would  take  advantage  of  the  immediacy  of  the  communication  which 
included  the  facial  expression  and  body  language  nuancing  between  all  participants  as  a community  exercise. 
Current  two-way  VC  technology  is  now  widespread  enough  with  offerings  such  as  Skype  (Microsoft), 
Facetime  (Apple)  and  others,  that  it  is  no  longer  seen  as  something  impossible  to  use  by  the  general  public. 
Although  these  systems  are  used  mostly  for  one-to-one  conversations,  many  systems  are  available  for  many- 
to-many  or  multipoint  VC,  and  are  being  used  in  the  higher  education  sector  as  a possible  solution  to  address 
a large  part  of  the  student  isolation  concerns  (Goldrick,  O’Higgins  Norman,  2012). 

This  brought  on  another  concern,  that  of  necessary  flexibility.  One  of  the  perceived  advantages  of  online 
learning  using  the  more  traditional  asynchronous  communications  models  is  that  students  can  work  on  their 
own  schedule,  at  their  own  pace.  Proposing  the  weekly  scheduled  video  meetings  seemed  to  seriously  reduce 
this  flexibility.  Creating  a model  that  would  strike  an  appropriate  balance  between  this  desired  flexibility  and 
the  need  for  opportunities  for  direct  real-time  interactions  while  maintaining  the  advantage  of  doing  all  this 
from  home,  no  matter  where  that  may  be,  became  a prime  objective. 

Moving  to  a mobile  virtual  platform  also  necessitated  the  adoption  of  a pedagogical  or  learning  model 
that  is  radically  different  from  that  used  in  most  traditional  classrooms  as  the  learners  will  be  expected  to 
access  Internet-based  documents  in  a wide  variety  of  formats  and  virtual  environments  rather  than  confining 
themselves  to  a specific  textbook  or  classroom  set  of  paper-based  documents.  An  online  problem  based 
learning  curriculum  development  model  (Savin-Baden,  2007)  was  adopted  as  it  allowed  for  the  incorporation 
of  problem  scenarios  or  contexts  within  which  learners  could  identify  their  own  problems  and  work  towards 
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creating  solutions,  thereby  learning  about  the  content  while  actively  engaging  with  it.  All  of  the  technological 
tools  required  for  this  type  of  work  are  available  for  mobile  device  platforms. 

It  has  also  often  been  stated  that  for  a great  variety  of  reasons,  many  of  today’ s post-secondary  learners 
tend  to  balance  studies  with  work,  family  and  other  time  constraints.  In  addition,  the  reality  of  spending  a 
substantial  amount  of  time  commuting  between  these  activities,  or  even  at  locations  away  from  home,  means 
that  there  would  be  a great  advantage  to  having  as  much  of  the  planned  learning  activities  designed  in  a 
manner  as  to  be  accessible  on  mobile  devices  from  anywhere,  anytime. 

For  these  reasons,  a Bachelor  of  Arts  in  Adult  Education  and  Digital  Technology  was  designed,  formally 
proposed  and  implemented.  It  was  to  be  offered  completely  online,  with  a dominant  real-time 
videoconferencing  component  and  a pedagogical  framework  that  would  foster  the  development  of 
meaningful  teacher-learner  and  learner-learner  relationships  as  well  as  taking  advantage  of  any  and  all 
opportunities  of  access  to  information  and  experts  in  the  domain  being  studied.  This  paper  discusses  some  of 
the  initial  stages  in  the  designing  and  implementation  process  as  well  as  some  of  the  concerns  that  have 
emerged  along  the  way. 


2.  PROGRAM  FRAMEWORK 

In  the  initial  documents,  intending  for  every  aspect  of  this  program  to  fall  within  all  university  regulations,  a 
typical  36  hour  (3  credit)  course  was  designed  to  be  articulated  in  12  weekly  modules  each  including: 

• a total  of  3-4  (approx.  12  minutes  each)  video  clips  publically  available  online  (ie.:  YouTube).  This 
was  to  represent  the  “lecture”  part  of  a traditional  course  and  was  intended  to  avoid  the  perceived 
necessity  of  such  lecturing  in  the  videoconferencing  portion  of  the  course.  The  video  clips  were 
initially  to  be  written  by  a content  expert,  (i.e.,  professor  from  our  own  Faculty  or  a recognised 
expert  from  another  institution).  These  have  since  mostly  been  produced  either  by  the  professors 
themselves  or  by  media  specialists.  These  were  then  to  be  installed  on  the  institution’s  video 
repository  or  simply  on  some  public  site  such  as  YouTube. 

• 60  minutes  of  synchronous  group  activities  in  Adobe  Connect  (videoconferencing)  moderated  by 
one  teaching  assistant  for  each  30  students.  These  could  include  planned  discussions  on  specific 
questions,  teamwork  on  predesigned  problems,  or  any  activity  that  would  foster  collaborative 
knowledge  construction.  The  synchronous  group  activities  in  the  videoconferencing  platform  are 
planned  in  detail  by  the  professor  (content  expert)  with  the  explicit  opportunity  to  consult  with  other 
professors  in  the  program  or  with  the  institution’s  instructional  designers.  The  online  activities  are 
then  run  by  the  TA  on  a weekly  basis.  By  scheduling  the  different  groups  at  different  times  of  the 
day  and  of  the  week,  the  courses  are  able  to  accommodate  students  from  a wide  variety  of  time 
zones 

• Equivalent  of  one  hour  of  work  to  be  done  online  asynchronously  (i.e.:  Blackboard  discussion 
forum,  wiki  entries,  etc.).  This  is  to  allow  for  students  to  engage  in  a more  reflective  set  of  activities 
where  time  can  be  taken  to  express  thoughts  or  ideas  at  one’s  individual  rate,  without  the  pressure  of 
the  immediate  discussion.  The  final  element  of  each  weekly  class  includes  many  other  online 
activities  such  as  forum  discussions,  self -directed  learning  activities,  etc.  as  deemed  appropriate  for 
the  course.  Although  this  covers  the  basic  weekly  three-hour  commitment  to  the  class,  it  is  to  be 
noted  that  additional  reading  or  other  course  work  is  usually  expected  from  the  student. 

In  order  to  meet  the  requirement  of  reaching  the  students  wherever  they  maybe  at  anytime,  the  technical 
systems  were  chosen  with  the  additional  characteristic  of  necessarily  being  able  to  run  on  most  mobile 
devices  such  as  the  smartphones  and  tablets  supported  by  the  dominant  mobile  platforms,  i.e.,  iOS,  Android 
and  BlackberrylO. 


3.  IMPLEMENTATION 
3.1  Online  Pedagogical  Model 

From  this  initial  design,  the  group  involved  in  the  program  made  use  of  a digital  space  that  currently 
includes:  1)  video  clips  publically  posted  on  YouTube  to  present  concepts  and  ideas,  2)  email  and  fde 
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repository  capabilities  as  provided  by  a Learning  Management  System  (LMS),  3)  synchronous  virtual 
audio/video  conferencing  conducted  using  Adobe  Connect®  for  tutorial  discussions  and,  4)  a wide  variety  of 
Open  Educational  Resources  (OER),  such  as  Wiki’s,  Discussion  Forums  and  Blogs  for  assignments  and 
tasks. 

Most  of  the  applications  comprising  this  digital  space  can  be  accessed  using  mobile  devices  of  all  kinds, 
thus  providing  systems,  structures,  and  processes  that  facilitate  ubiquitous  learning.  The  digital  space  is  used 
to  define  the  context  of  a ‘completely’  online  (using  synchronous,  as  well  as  asynchronous  digital  tools) 
program  in  that  all  interactions  with  students  in  a formal  ‘course’  setting  will  occur  within  the  space  to  ensure 
equity  of  access  and  experience  to  all  students  regardless  of  geographic  or  temporal  location. 

The  model  designed  for  the  development  of  courses  within  the  program  was  based,  to  a large  extent,  on 
the  work  of  Anderson  (2007)  and  Savin-Baden  (2007).  The  model  details  that  learning  within  the  program 
occurs  where  the  sphere  of  social  presence  (the  availability  for  interaction  with  others)  interacts  with  the 
sphere  of  cognitive  presence  (the  availability  for  interaction  with  ideas  and  theories,  as  well  as  the  creation  of 
knowledge). 

Social  presence  within  the  program  occurs  within  the  context  of  an  online  community  of  inquiry  that  was 
welcoming  to  a wide  diversity  of  learners.  It  was  understood  that  the  community  could  not  be  structured  as 
most  physical  classrooms  are,  with  information  dispensers  and  information  gatherers,  since  classrooms  in  this 
program  are  situated  in  the  Internet,  which  provides  access  to  more  information  than  any  single  teacher  could 
ever  hope  to  provide.  Since  there  was  no  need  to  provide  material  to  the  learners  and  in  order  to  give  the 
learners  opportunities  to  investigate  issues  of  personal  import  it  was  decided  to  pursue  a learning  model  that 
was  as  horizontally  flat  as  possible.  All  members  of  the  community,  inclusive  of  students,  teaching  assistants 
and  professors,  would  be  involved  in  learning,  critiquing,  assessing  and  evaluating.  In  the  online  learning 
environment,  the  distinetions  of  power,  the  barriers  between  the  ‘haves’  and  the  ‘have  nots’,  and  the 
obstacles  of  authority  or  expertise  were  not  as  important  as  promoting  communal  collaborative  inquiry 
allowing  students  to  freely  and  safely  exchange  criticisms,  views,  and  opinions. 

According  to  Anderson  (2007),  the  cognitive  presence  encourages  students  to: 

• approach  problems  creatively  and  strategically, 

• actively  seek  out  sources  of  information, 

• identify  and  address  bias,  prejudice,  and  privilege, 

• manage,  analyse  and  synthesize  large  quantities  of  information,  and 

• formulate  and  defend  personal  views  and  positions. 

Students  in  the  BA  program  access  course  ‘lectures’  in  the  form  of  publically  available  video  files  posted 
to  YouTube  using  a Creative  Commons  - Attribution  licence.  YouTube  was  chosen  as  the  publication  site 
primarily  due  to  its  availability  to  most  individuals  with  apps  provided  for  all  of  the  major  mobile  platforms. 
Publically  publishing  video  clips  dealing  with  educational  material  on  YouTube  not  only  establishes  Open 
Educational  Resources  (OER),  it  simultaneously  plays  an  outreach  function  allowing  those  not  registered  in 
the  courses  to  access  the  material. 

Within  the  B.A.  courses,  the  institutional  learning  management  system  (LMS)  is  used  as  a common  email 
and  file  repository  system.  All  program  students  are  registered  into  an  additional  ‘course’  that  is  set  up  within 
the  LMS.  This  allows  for  posting  of  announcements  and  administrative  messages  that  are  sent  directly  to  the 
students. 

Within  small  tutorial  groups,  students  discussed  their  particular  understandings  of  the  situation  portrayed 
in  the  video  clips,  eventually  agreeing  on  a specific  question  or  questions  that  they  would  collaboratively 
investigate,  identifying  resources  and  knowledge  that  was  possessed  or  required  in  order  to  create  answers  to 
the  questions  that  were  posed  (vanOostveen,  Desjardins  & Bullock,  2010). 

In  this  program,  collaborative  learning  occurs  primarily  through  the  use  of  synchronous  technologies, 
such  as  Adobe  Connect  and  Skype,  in  an  environment  in  which  the  cognitive  presence,  social  presence,  and 
digital  space  intersect;  an  environment  in  which  "members  of  a community  of  inquiry  [. . .]  construct  meaning 
though  sustained  communication"  (Anderson,  2008).  Learners  are  encouraged  to  nurture  the  development  of 
valuable  social  and  cognitive  skills  that  are  essential  to  the  establishment  of  a critical  learning  community.  A 
learning  community  relies  on  the  process  of  peer  review  to  ensure  that  a certain  standard  of  rigour  and  quality 
is  maintained  (Wenger,  2000).  Students  in  the  B.A.  program  learn  to  adopt  characteristics  of  a community  of 
learners  in  order  to  determine  the  strength  of  warrants  for  knowledge  claims.  Longino  (1994)  identifies  four 
conditions  that  a community  of  practitioners  must  meet  if  consensus  is  to  count  as  knowledge  rather  than 
mere  opinion. 
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1 . There  must  be  publicly  recognized  forums  for  criticism. 

2.  There  must  be  uptake  of  criticism. 

3.  There  must  be  publicly  recognized  standards  for  evaluation  of  theory  and  practice. 

4.  There  must  be  equality  of  intellectual  authority. 

Students  within  the  program  are  regularly  required  to  engage  in  academic  critiques  of  their  work  and  the 
work  of  their  colleagues. 

The  establishment  of  a learning  community  requires  both  asynchronous  and  synchronous  technologies. 
Synchronous  online  learning  has  been  reported  as  being  more  social  in  nature  than  asynchronous  online 
learning  and  it  avoids  frustration  by  allowing  for  conversations  in  real  time.  Two-way  videoconferencing 
offers  access  to  all  the  non-verbal  communication,  all  of  which  provides  for  greater  nuancing  of  the 
interactions.  In  addition,  social  networking  apps  such  as  Twitter  and  many  blogging  sites,  are  growing  in  use 
within  the  program,  and  are  intended  to  foster  a different  mode  of  asynchronous  discussion.  Many  of  these 
applications  have  associated  mobile  apps  or  are  available  through  mobile  browsers. 

3.2  Curriculum  Development  Model 

Content  experts  from  within  UOIT’s  Faculty  of  Education,  other  Faeulties  and  from  beyond  the  university 
were  contracted  to  develop  the  course  outlines,  video  clips,  course  assignments/tasks,  rubrics  and  any  other 
activities  in  order  to  create  an  online  environment  within  which  the  students  could  learn.  The  developers 
followed  a curriculum  development  model  designed  to  promote  problem-based  learning  (PEL)  (Savin-Baden, 
2007)  and  constructivist  learning  orientations  (Vygotsky,  1978). 

Each  course  contains  several  opportunities  to  use  collaborative  PEL  processes  to  learn  about  content.  This 
is  typically  done  using  the  video  clips  posted  to  YouTube  to  present  students  with  a situation  or  a context 
representing.  PEL,  as  used  in  the  B.A.  program,  is  defined  as  ‘a  curriculum  model  designed  around  real  life 
problems  that  are  ill  structured,  open  ended  or  ambiguous’  and  it  is  suggested  that  ‘PEL  engages  students  in 
intriguing,  real  and  relevant  intellectual  inquiry  and  allows  them  to  learn  from  these  life  situations’  (Fogarty, 
1997).  Problems  are  identified  by  the  students  when  they  bring  their  own  particular  perspectives  to  bear  on 
the  presented  situation.  Once  a problem  or  set  of  problems  have  been  chosen  by  a group  of  students,  over  the 
course  of  the  next  few  weeks  they  will  meet  virtually,  using  Skype  or  a general  Adobe  Connect  room  that  has 
been  set  up  for  all  students  taking  B.A.  courses.  The  general  room  is  available  to  all  students,  TAs  and 
professors  24hrs  per  day,  7 days  per  week. 

The  video  clips  are  formatted  very  precisely  using  a pattern  laid  out  in  the  curriculum  development 
document  requiring  the  incorporation  of  3-5  analysis  questions  at  the  beginning  of  every  clip.  Analysis 
questions  are  designed  to  invite  the  viewer  to  break  down  the  remainder  of  the  video  clip  into  component 
parts  in  order  to  identify  embedded  problems  (inductive  processes).  A further  3-5  synthesis  questions  are 
posed  at  the  conclusion  of  each  video  clip.  Synthesis  questions  are  designed  to  invite  the  viewer  to  compile 
information  gleaned  from  the  video  clip  content  to  propose  solutions  (deductive  processes).  The  video  clips 
then  become  instigations  for  the  discussions  that  follow  in  the  tutorial  sessions  conducted  using  Adobe 
Connect. 

The  structure  of  the  tutorials  tends  to  be  most  effective  if  there  is  a strong  tie  between  the  video  clip 
portion  of  the  courses  and  the  tutorial  sessions.  As  the  sessions  are  usually  facilitated  by  Teaching  Assistants 
(TAs),  it  was  vital  that  a strong  relationship  was  established  between  the  instructor  and  the  TA  in  a course. 
TAs  and  professors  function  as  facilitators,  rather  than  disseminators  of  information,  during  the  tutorial 
sessions.  De  Grave,  Dolmans  & van  der  Vleuten  (1999)  state  that  the  facilitation  role  should  “scaffold 
student  learning  in  order  to; 

• simulate  the  elaboration  of  information  and  ideas 

• guide  the  learning  process,  including  stimulation  of  reflection  on  the  learning  process 

• stimulate  the  creation  and  integration  of  knowledge 

• stimulate  and  support  student  interaction  and  individual  accountability”  (p.901) 

Tasks  and  activities  which  are  designed  to  provide  synthesis,  evaluation  and  creation  experiences  are 
assigned  to  students  while  they  are  engaged  in  the  courses.  While  there  is  a fair  amount  of  variety  in  the  types 
of  tasks  that  are  assigned,  such  as  the  production  of  learning  objects,  video  ‘papers’,  and  briefing  notes,  these 
are  primarily  collaborative  in  nature,  requiring  students  to  meet  virtually  using  a number  of  technologies  such 
as  email,  Twitter,  Google  Drive,  YouTube,  ScreenCast,  etc.  Course  TAs  meet  virtually  with  the  students  on  a 
regular  basis  in  order  to  support  this  work. 
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4.  CONCLUDING  REMARKS 

At  the  time  of  this  writing,  students  in  the  B.A.  program’s  first  class  are  entering  their  second  term.  There  are 
approximately  20  students  registered  in  the  program,  approximately  half  of  these  are  registered  in  4 or  5 
courses  per  term  and  the  remainder  are  registered  in  1 or  2 courses  as  fits  their  personal  schedules  and 
workloads.  While  this  is  not  a large  number  of  students  for  the  inaugural  year,  it  has  been  reported  that  this  is 
an  extraordinarily  good  response  for  a program  that  bridges  between  college  and  university  experiences. 
Additional  students,  4 or  5 per  course,  from  other  faculties  in  UOIT  have  joined  the  program  by  taking 
individual  courses  as  electives  to  augment  their  studies  in  other  degree  programs. 

Most  of  the  students  who  are  registered  in  the  program  are  working  full  time  and  many  have  families  that 
need  part  of  their  daily  attention.  A wide  diversity  of  backgrounds  is  represented  with  students  coming  from 
security  services,  fine  arts,  and  various  commercial  interests.  Several  of  the  students  are  employed  as 
instructors  by  community  colleges  in  Ontario.  These  students  are  required  to  obtain  university  degrees  and 
the  B.A.  program  provides  the  flexibility  to  do  this  while  continuing  to  be  employed. 

Over  the  course  of  the  first  term,  teaching  staff  (professors  and  TA’s)  and  students  were  polled  every  few 
weeks.  Two  questions  were  asked:  1)  What  do  you  like  about  this  program  so  far?  Why?  and,  2)  What  would 
you  like  to  change/modify  about  the  program  so  that  it  better  suits  your  needs?  Why?  Approximately  Vi  of 
the  teaching  staff  and  student  population  responded  to  each  survey 

Anecdotal  responses  to  the  first  question  reflected  a positive  reaction  to  the  flexibility  of  the  program  in 
terms  of  time,  as  the  only  synchronous  portions  are  the  tutorials.  Some  students  mentioned  that  a warm 
collaborative  learning  environment  had  been  established  allowing  them  to  feel  connected  to  classmates  and 
the  learning  community  as  a whole.  PBL  was  singled  out  as  being  a new  way  to  learn  which  required  some 
adjustment  but  was  appreciated  none  the  less. 

The  comments  posted  in  response  to  the  second  question  reflected  the  transitioning  that  was  required  of 
all.  Most  of  the  statements  reflected  challenges  using  all  of  the  digital  technologies.  Some  students  noted 
difficulties  in  synchronizing  schedules  with  others  within  groups  and  a struggle  to  adjust  to  the  demands  of  a 
university  program  which  requires  a lot  of  reading,  collaborative  group  work  and  synthesis  and  evaluation  of 
concepts.  One  student  complained  that  the  tutorial  sessions  were  not  long  enough  as  the  discussions  are  often 
cut  short.  An  instructor  also  noted  that  since  the  students  are  included  in  the  discussions  about  customizing 
the  learning  environment  the  modifications  of  the  courses  were  being  done  continuously  reflecting  an 
individualized  curriculum,  or  perhaps  making  significant  strides  towards  this  goal. 


5.  COURSE  DEVELOPMENT  COMMENTS 

10  courses  have  been  developed  to  date  for  the  B.A.  program.  Five  of  these  were  implemented  over  the  past 
term,  with  the  remaining  five  being  initiated  in  the  winter  2013  term.  Some  of  the  courses  are  foundational  to 
the  program.  For  instance,  in  the  fall  term,  the  Foundations  of  Digital  Teaching  and  Learning  Technologies 
course  was  offered,  providing  students  a framework  that  can  be  used  to  analyse  teaching  and  learning 
technologies.  Additional  courses  regarding  the  Foundations  of  Adult  learning  and  Psychological  Foundations 
of  Digital  Technologies  will  be  implemented  in  the  winter  term.  Other  offerings  include  courses  in  Problem 
Based  Learning,  Creating  Digital  Tools,  Culture  and  Digital  Technologies,  Digital  Technologies  and 
Advanced  Teaching  Methods,  Online  Learning,  Professional  Writing,  as  well  as  a course  about  Serious 
Games  and  Simulations.  Next  year  there  are  plans  to  develop  an  additional  9 courses,  ranging  from  Digital 
Communication  Technologies,  Graphic  Design,  Information  Literacy,  Workplace  Learning  and  a year-long 
double  course  that  requires  students  to  design  and  implement  an  applied  research  project  resulting  in  a brief 
thesis.  In  the  following  2 years,  the  remainder  of  the  courses  will  be  developed  and  brought  online  when  they 
are  required  for  the  Direct  Entry  registrants  who  will  register  in  the  program  directly  from  high  school.  In  the 
past  year,  and  in  the  upcoming  years,  there  are  several  courses  that  will  be  and  have  been  developed  by 
professors  in  other  faculties.  It  is  likely  that  these  courses  will  be  cross-listed  for  students  in  other  programs 
so  that  the  university  as  a whole  can  benefit  from  the  development  of  these  fully  online  courses,  providing 
alleviation,  for  those  who  prefer  the  convenience,  of  the  space  constraints  currently  experienced. 

Professors  who  are  interested  in  course  development  work  fulfil  a contract  with  the  UOIT  Teaching  and 
Learning  Centre.  Comments  from  course  developers  describe  the  extreme  amount  of  work  and  writing  that  is 
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required  to  produce  the  course  outlines,  tasks  and  video  clips.  One  professor  estimated  that  the  video  clip 
scripts,  which  are  required  in  order  to  transcribe  the  audio  components  for  closed  captioning  on  YouTube, 
comprised  in  excess  of  40,000  words  or  a sizable  book.  Another  professor  expressed  an  appreciation  for  the 
level  of  detail  that  was  provided  in  the  curriculum  development  model  but  as  it  did  not  provide  a prescription 
for  the  course  that  was  being  built,  the  professor  found  herself  engaged  in  a problem  based  learning  scenario 
and  she  was  able  to  produce  a course  that,  in  a paraphrase  of  her  words,  will  be  engaging  and  demanding  for 
the  students.  Other  challenges  lie  in  the  area  of  keeping  the  video  clips  short,  relevant  and  engaging. 
Obtaining  copyright  permissions  is  another  issue  that  has  required  due  diligence  on  the  part  of  the  program 
administration  and  library  staff. 

In  December  of  2012,  a full  day  workshop  was  held  for  more  than  20  teaching  staff  using  the  services 
provided  through  the  Adobe  Connect  general  room.  The  program  for  the  workshop  included  sessions  on; 
Managing  the  Culture,  which  concentrated  on  understanding  the  technology/pedagogy  relationship  and 
process  of  change;  BA  Graduate  Career  Aspirations,  which  focussed  on  determining  what  are  the  target 
competencies  for  graduates  of  the  program;  and  finally.  Assessing  the  Program/Students.  The  main  function 
of  the  workshop  was  to  provide  opportunities  for  all  to  discuss  these  issues  so  that  a set  of  common  views 
could  begin  to  develop.  Teaching  staff  also  experienced  limited  project  based  learning  activities  and  were 
able  to  better  appreciate  it,  particularly  if  it  was  new  to  them. 

A number  of  challenges  exist  across  the  university  when  contemplating  moving  to  an  mobile  online 
model  of  teaching  and  learning.  In  order  to  determine  what  these  challenges  are,  a gap  analysis  is  being 
conducted.  It  is  hoped  that  this  process  will  be  able  to  identify  the  discrepancies  between  the  level  of  services 
that  are  not  yet  available  but  are  required  for  fully  online  learners  to  take  their  place  in  the  university  student 
body  without  ever  physically  being  present  on  campus.  Some  of  these  challenges  come  from  those  who 
express  serious  reservations  about  the  level  of  academic  rigour  that  is  possible  in  online  learning 
environments.  Other  challenges,  such  as  faculty  development,  technology  support,  resource  availability  and 
programming  issues  have  already  been  alluded  to.  Others  such  as  a rationalized  fee  structure  and  the 
implementation  of  a comprehensive  orientation  program  have  yet  to  be  resolved.  Promotion  of  the  program 
beyond  the  borders  of  the  province  and  the  country  when  all  of  the  recruitment  efforts  to  date  have  been 
focussed  on  face-2-face  meetings  also  offers  a considerable  test.  It  is  hoped  that  the  use  of  social  media  and 
Internet  advertising  will  be  of  some  value  in  getting  the  message  out  regarding  the  program.  Finally,  building 
and  implementing  a course  and  program  evaluation  process  that  recognizes  the  various  constraints  inherent  in 
online  teaching  and  learning,  along  with  on-going  program  review  processes,  will  be  vital  to  the  viability  of 
this  type  of  program. 

While  the  number  of  outstanding  challenges  can  be  daunting,  the  overall  promise  of  moving  into  an 
online  teaching  and  learning  environment,  particularly  when  it  is  premised  on  the  use  of  synchronous  digital 
technologies  and  includes  opportunities  to  employ  mobile  devices  providing  anytime,  anywhere  access,  is 
positive.  As  can  be  seen  from  this  project,  centred  on  the  creation  of  the  B.A.  in  Adult  Education  and  Digital 
Technology  program  at  UOIT,  flexibility  of  thought  and  planning  provides  an  answer  for  the  physical 
constraints  that  currently  hamper  the  university  while  simultaneously  opening  a door  to  a possible  revisioning 
of  educational  and  learning  processes. 

Finally,  concerns  have  been  raised  on  the  issue  of  service  provisioning  for  online  students  as  being  totally 
different  to  regular  on-site  students.  From  the  perspective  of  working  within  fully  online  programs,  it  is 
possible  to  forget  that  others  may  not  think  that  attending  class  from  your  office  or  home  is  normal  and 
therefore,  this  kind  of  information  is  now  flowing  up  to  many  institutional  departments  to  find  ways  to  better 
serve  these  new  online  students.  Some  of  the  issues  currently  under  study  are: 

1.  Tuition  and  auxiliary  fee  structures  for  online  students  (must  be  transparent  in  the  application  of 
charges  and  non-residential  fees  should  not  be  substantially  different  from  residents  as  there  are  few 
additional  costs) 

2.  Establishment  of  an  online  learning  community 

3.  Inclusion  of  online  students  in  student  life  and  government 

4.  Orientation  programs  to  invite  students  into  each  term 

5.  Ongoing  communication  between  units  of  the  university  which  are  responsible  for  the  myriad  of 
details  regarding  the  program 

6.  Programming,  including  cross-listing  across  programs  and  faculties,  maximizing  access  to  courses 
for  all  students 

7.  Registration  services  and  program  administration  support 
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8.  Technical  support  for  the  IT  infrastructure  that  is  developing 

9.  Faculty  development  creating  links  between  facilitation  and  resources 

10.  Library  support,  particularly  in  the  area  of  open  educational  resources 

1 1 . Specific  online  course  and  program  evaluation  processes  that  are  separate  from  those  used  for  f2f 
classes 

Solving  these  issues  represents  some  of  the  next  steps  in  institutionalising  online  learning  that  is  mobile 
ready,  and  solving  these  will  allow  us  to  move  towards  the  next  level  of  mobile  learning  in  the  Faculty  of 
Education. 
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ABSTRACT 

International  Standards  in  High  and  Open  and  Distance  Education  are  used  for  developing  Open  Educational  Resources 
(OERs).  Current  issues  in  e-learning  community  are  the  specification  of  learning  chunks  and  the  definition  of  describing 
designs  for  different  units  of  learning  (activities,  units,  courses)  in  a generic  though  expandable  format.  Developing 
educational  content,  courseware  and  pedagogical  tools  to  support  Subject  Matter  Experts  in  their  educational  practice  is  a 
challenging  task  for  the  Instructional  Design  Eield.  This  paper’s  aim  is  to  introduce  a user  oriented  design  methodology 
for  Open  and  Distance  Learning  (ODL)  activities  orchestrating  pedagogical  tools  so  as  to  develop  e-learning  activities  in 
Hellenic  Open  University  (HOU)  by  the  use  of  Boundary  Objects  (BOs). 


KEYWORDS 

Participatory  Design,  Communities  of  Practice,  Boundary  Objects,  Learning  Design,  Design  Research,  Empirical 
Research 


1.  INTRODUCTION 

With  the  advent  of  technological  tools  and  a plethora  of  learning  theories  provided,  course  designers  and 
Subject  Matter  Experts  need  guidance  in  producing  learning  activities  which  take  account  the  new  reality 
(complexity  of  technological  tools  provided,  variation  in  students’  profile,  culture  of  subject  domain)  and 
maximize  the  potential  technologies  offer.  The  design  problem  arising  out  of  that  educational  reality  is  about 
capturing  and  representing  practice  in  Higher  Education  (HE)  institutions  and  providing  “scaffolding”  or 
support  in  practitioners  for  designing  learning  activities  which  draw  on  good  practice  and  make  effective  use 
of  tools  and  pedagogies,  not  necessarily  in  a linear  mode.  Sharing  and  reuse  of  educational  material  needs  to 
be  based  on  these  attributes  of  learning  design:  a)  to  be  based  on  sound  pedagogical  principles,  b)  to  promote 
sharing  of  expertise,  c)  to  support  the  community  through  available  support  services  (Walker  and 
Masterman,  2010). 

Learning  Design  is  not  understood  uniformly,  with  perceptions  ranging  from  misconstruction  to  highly 
developed  understanding:  teachers  need  technical  training  on  how  to  use  the  tools  but  also  they  need  training 
to  help  them  understand  how  to  use  the  tools  to  achieve  pedagogical  goals  (Bower  and  Wittmann,  2010).  In 
this  case  basic  research  aim  has  been  to  involve  practitioners  in  the  design  of  learning  activities  and  capture 
their  contribution  on  the  structures’  schematization  of  Open  and  Distance  Learning  activities  followed  by  the 
selection  of  the  appropriate  educational  content.  The  activity  definition  selected  in  this  research  approach  has 
been  the  one  viewing  learning  activities  in  relation  to  the  design  process  “as  a specific  interaction  of 
learner(s)  with  other(s)  using  specific  tools  and  resources,  orientated  towards  specific  outcomes”  (Beetham 
and  Sharpe,  2007).  The  rationale  has  been  to  structure  small  Communities  of  Practice  (CoPs)  as  a 
springboard  to  further  expand  the  future  development  of  online  larger  communities,  based  on  models  which 
offer  “scaffolding”  into  the  practice  of  sharing. 
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Most  of  the  current  instructional  design  models  are  spin-offs  or  variations  of  the  ADDIE  instructional 
design  model  (Peterson,  2003).  However,  one  commonly  accepted  improvement  to  this  model  is  the  use  of 
Rapid  Prototyping  (Fischer,  2011):  the  idea  of  receiving  continual  or  formative  feedback  while  instructional 
materials  are  being  created,  catching  design  problems  while  they  are  still  easy  to  fix.  Springboard  of  the 
design  rationale  of  the  specific  research  scheme  has  been  not  to  use  from  the  early  design  stages  a specific 
Instructional  Design  model,  but  use  instead  a methodology  open  for  allowing  the  emergence  and  iterative 
development  of  activity  schemata  and  pedagogical  tools  under  development.  A Learning  Design  ( Koper, 
2006  ) approach  has  been  adapted  as  a)  a means  of  eliciting  designs  from  academics  in  a format  that  can  be 
tested  and  reviewed  by  developers,  b)  a means  of  reusing  designs  of  activities,  c)  a means  of  tentatively 
supporting  practitioners  through  the  process  of  creating  new  learning  activities  . As  Conole  (2010)  has  stated 
the  key  facet  to  adapt  and  reuse  existing  learning  activities  has  been  to  abstract  the  essential  and  transferrable 
properties  of  learning  activities  in  generic  schemata,  expandable  however  in  more  complicated  forms.  As  a 
means  to  achieve  that  we  have  selected  the  Participatory  Design  (PD)  approach  as  an  instructional  strategy 
technique  and  the  use  of  Mediating  Artifacts  ( MA,  Conole  2010)  in  a Communities  of  Practice  (CoPs) 
format  as  means  of  developing  learning  activities’  structures  and  generic  pedagogical  tools,  used  by  the 
Hellenic  Open  University’s  practitioners.  There  has  been  an  effort  to  ground  Instructional  Design  Theory 
(IDT)  in  educational  practice,  to  examine  the  profession  as  a living,  changing  and  growing  practice  capturing 
elements  and  aspects  of  Tutors’  educational  experience  in  shaping  learning  activities’  structures  and 
pedagogical  tools  under  development.  In  providing  an  optimum  learning  experience  we  chose  a)  to 
continually  evaluate  the  learning  media  for  usability  and  effectiveness,  and  as  early  as  possible,  b)  to 
immediately  revise,  rewrite,  recreate,  or  add  to  the  learning  media  in  a cyclic  pattern.  Basic  aims  have  been  to 
understand  the  mechanisms  of  interaction  between  ( i)  the  world  of  design  and  users  and  ii)  between  loops  of 
CO  design  and  loops  of  uses  so  as  to  produce  guidelines  of  an  user  oriented  methodology  for  designing  and 
developing  learning  activities  and  supportive  pedagogical  tools  which  provide  scaffolding  to  practitioners. 
This  paper  is  structured  as  follows:  a short,  critical  presentation  of  the  Learning  Design  approach  is 
developed  followed  by  the  basic  characteristics  of  Rapid  Prototyping  and  Boundary  Objects  which  structured 
the  basis  of  the  proposed  methodology.  In  the  proceeding  sections  a description  is  presented  of  the  research 
schema  implementation  and  basic  components  of  the  first  generation  pedagogical  design  followed  by 
preliminary  data  comprising  an  example  of  the  methodology  introduced.  Conclusion  is  presented  in  section  4. 


2.  DEVELOPING  A USER  ORIENTED  DESIGN  METHODOLOGY  FOR 
E-LEARNING  ACTIVITIES 


2.1  The  Learning  Design  (LD)  Approach 

The  term  Learning  Design  (LD)  has  been  inextricably  linked  with  the  development  of  IMS -Learning  Design 
specification  (http://www.imsglobal.org/learningdesign/index.html)  aiming  at  providing  a means  of  formal 
representation  and  reuse  of  learning  sequences.  Since  then  LD  has  been  broadly  used  by  researchers  and 
developers  in  course  design  and  there  has  been  a shift  from  content  creation  and  presentation  to  focus  on 
learning  activities.  The  ADDIE  model  has  been  dominant  and  profoundly  influential  in  forming  educational 
practice  in  the  terms  of  designing  educational  courses  and  adaptive  systems  supporting  these  ( Conole  and 
Retails,  2010).  However,  it  has  been  criticized  of  a)  not  being  necessarily  leading  to  the  best  instructional 
solutions,  b)  not  providing  solutions  in  a timely  and  efficient  manner,  c)  not  taking  advantage  of  technologies 
that  allow  for  less  linear  approaches  to  Instructional  Design  (ID)  such  as  Rapid  Prototyping  (RP).  It  has  been 
argued  that  its  application  in  real  educational  settings  is  time  consuming  and  produces  boring  courseware 
(Dicketal,  2011). 
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2.2  Rapid  Prototyping  (RP)  and  Boundary  Objects  (BOs) 

The  Rapid  Prototyping  (RP)  process  involves  quickly  developing  a prototype  product  in  the  very  early  stages 
of  the  instructional  design  project  and  then  going  through  a series  of  rapid  tryouts  and  revision  cycles:  this 
design  technique  has  been  advocated  as  a means  of  producing  quality  instructional  materials  in  less  time  than 
is  required  when  more  conventional  instructional  design  techniques  are  employed.  Rapid  prototyping  models 
involve  learners  and/or  Subject  Matter  Experts  interacting  with  prototypes  and  instructional  designers  in  a 
continuous  review/revision  cycle  (Fischer  et  al,  2011,  Dick  et  al,  2012).  Developing  a prototype  is  practically 
the  first  step,  while  front-end  analysis  is  generally  reduced  or  converted  into  an  on-going,  interactive  process 
between  subject-matter,  objectives,  and  materials.  Basic  criteria  for  implementing  this  process  have  been  a) 
promoting  the  discussion  between  Practitioners,  Instructional  Designer,  Learners  in  a focused  way,  by 
concentrating  on  the  facts  and  the  results  ,rather  than  a priori  selected  and  defined  ID  models  and  learning 
theories,  b)  developing  shared  understanding  among  the  HOU  practitioners  and  instructional  designer 
involved  in  the  project  so  as  to  build  trust  and  create  a common  language  between  fields,  c)  making  the 
design  and  development  process  open  to  new  emerging  ideas,  d)  making  the  design  open  to  emerging  needs 
from  test  and  evaluation  phases,  e)  focusing  on  pedagogical  design  (teaching)  instead\of  course  materials 
preparation  and  technology.  The  rapid  prototype  creates  an  early  iteration  loop  that  provides  valuable 
feedback  on  technical  issues,  creative  treatment,  and  effectiveness  of  instruction. 

CoPs  consist  of  practitioners  who  work  as  a community  in  a certain  domain  undertaking  similar  work 
(Fischer,  2011).  A community  of  practice  has  many  possible  paths  and  many  roles  (identities)  within  it  (e.g., 
leader,  scribe,  power-user,  visionary,  and  so  forth).  Boundary  objects  (Fischer,  2011)  are  externalizations  of 
ideas  that  are  used  to  communicate  and  facilitate  shared  understandings  across  spatial,  temporal,  conceptual, 
or  technological  gaps:  as  intermediary  productions,  they  result  from  interactions  between  designers’  and 
users’  world,  based  on  rules  identified  by  the  designers:  utilization  of  these  artifacts  in  the  users’  activity 
constitutes  an  opportunity  for  testing  the  hypothesis  of  designers,  revealing  their  consequences  and  possibly 
enriching  or  shifting  them.  These  objects  incorporate  the  knowledge  coproduced  during  the  design  process: 
uses  in  real  practice  improve  the  validity  of  this  produced  knowledge.  Boundary  objects  play  an  important 
role  in  the  emergence  of  both  socio-technical  compromises  and  mutual  learning  processes.  In  order  to  play 
this  role  three  characteristics  are  essential:  a)  robustness:  boundary  knowledge  incorporate  all  the  knowledge 
(Theoretical  and  Actionable)  produced  during  the  design  process  and  allow  for  their  interactions,  b) 
openness:  the  use  of  generic,  plastic,  reconfigurable  objects  allows  for  their  adaptation  to  different  purposes 
and  needs,  c)  simulation/visualization  capacity:  visual  representations  as  used  in  this  case  are  a basis  for 
practitioners’  -designers’  communications  and  enrich  the  knowledge  creation  process.  End  User 
Development  (EUD)  is  necessary  as  it  is  impossible  to  design  artifacts  (software  systems,  learning 
environments  ) at  design  time  for  all  the  problems  that  occur  in  use  time. 


3.  IMPLEMENTATION 

Research  shows  that  tutors  prefer  to  use  models  as  an  inspiration  for  creating  learning  designs  that  suit  their 
own  style  and  context  rather  than  simply  copying  the  shared  samples  (Walker  and  Masterman,  2010)  . A 
major  obstacle  to  teacher  adoption  of  learning  designs  is  an  insufficient  level  of  pedagogical  understanding 
required  to  make  use  of  resources  (Cameron,  2010).  Basic  aim  has  been  to  capture  HOU  Lecturers’ 
experience  of  educational  practice  and  allow  for  emergence  of  reusable  concepts  and  educational  content  in 
Micro  level  (activity)  to  Meso  level  (module)  and  Macro  level  (Thematic  Unit)  of  (4)  Thematic  Units.  (14) 
HOU  Lecturers  participated  in  the  conducted  research,  as  well  as  (200)  adult  students  of  both  Undergraduate 
and  Postgraduate  Courses.  Focus  Groups  sessions  have  been  used  as  a context  for  supporting  the  design  of 
learning  activities  and  actual  testing  of  LAMS  (http://lams.org)  learning  activities  in  adult  HOU  students  has 
taken  place  during  October  and  November  of  2012.  Basic  pedagogical  guidelines  have  been  adopted  to 
produce  generic  learning  designs  through  the  use  of  pedagogical  planning  tools:  generic  activity  templates 
have  been  produced  and  educational  content  in  the  form  of  Learning  Objects  has  emerged  out  of  the  process. 
Basic  aim  has  been  to  allow  for  iterative  cycles  of  design  with  in  parallel  development  of  both  pedagogical 
planning  tools  used  and  activity  schemata  produced  by  the  HOU  Tutors.  The  designed  activities  have  been 
oriented  towards  learning  trends  such  as  assessment  based  learning,  cooperative  learning,  self  regulated 
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learning  (Beethan  and  Sharpe,  2007):  their  generic  pedagogical  characteristics  emerged  out  of  the  tutors’ 
design,  they  are  however  expandable  in  richer  formats  and  more  complex  forms.  The  structural  components 
of  the  methodology  used  have  been  intentionally  selected  to  apply  in  an  array  of  subject  domains:  HOU 
numbers  (203)  Thematic  Units  in  (4)  Schools:  Humanities,  Social  Sciences,  Sciences  and  Technology, 
Applied  Arts.  The  LAMS  platform  has  been  used  experimentally  in  order  to  assess  students’  and  teachers’ 
attitudes  on  using  it  as  an  ICT  tool  in  their  educational  and  learning  practice.  However,  the  approach 
implemented  can  be  mapped  in  other  LMS  platforms  such  as  Moodle.  Whereas  LAMS  is  designed  to  operate 
more  at  the  level  of  individual  lessons,  Moodle  is  designed  to  structure  courses  (Bower  & Whittmann,  201 1). 
LAMS  more  tightly  defines  the  sequence  with  which  resources  and  activities  are  accessed  whereas  Moodle 
offers  more  student  control  over  the  order  in  which  resources  and  tools  are  used.  There  are  limitations  in  each 
case,  however  there  is  also  the  ability  to  implement  Moodle  learning  activities  within  LAMS.  The  produced 
representations  out  of  the  design  process  have  been  a)  learning  sequences  in  LAMS,  b)  conceptual  maps  of 
the  educational  content  used,  c)  paper  artifacts  representing  the  structure  of  learning  activity,  d)  verification 
of  the  structure  of  the  pedagogical  planning  tool  designed  and  used,  the  Design  for  Pedagogy  Tool  (D4P). 
Practice  focused  representations  have  been  used  ( eg  case  studies,  lesson  plans  and  patterns),  conceptual 
representations  (eg  mind  maps),  abstract  representations  eg  (vocabulary)  (Conole,  2010). 

3.1  Methodology  Applied 

E-Co-Me  T Lab  (http://www.eeyem.eap.gr  ) is  a supporting  organization  of  HOU  courses  providing 
innovations  in  design  of  educational  methodologies  and  educational  content  production.  A design  research 
methodology  (Fischer  et  al,  2011)  has  been  implemented,  involving  HOU  Lecturers  and  the  E-Co-Me  T Lab 
ID  Team  in  producing  mediating  artifacts  for  (4)  TU  of  HOU  that  have  been  used  so  as  to  design  and  develop 
learning  design  activities  using  LAMS.  The  design  research  approach  has  been  used  so  as  to  form 
prototheory,  explore  concepts  and  processes  so  as  to  further  develop  in  the  future  the  initial  design  in  iterative 
cycles  (Fischer,  201 1).  Practitioners  used  a)  flashcards  to  arrange  the  infrastructure  of  a Learning  Activity  in 
LAMS,  b)  diagrammatic  representation  of  the  educational  content  used  for  the  activity  design,  c)  Bloom’s 
taxonomy  so  as  to  produce  expected  Learning  Outcomes  (LOs)  for  their  activities.  Basic  steps  of  the  activity 
design  process  are  presented  in  Figure  1 : 


Define 
education 
al  content 

Organise 
Education 
al  Content 

Define 

Learning 

Outcomes 

Define 

activity 

structure 

Test  activity 

Redesign 

activity 

Figure  1.  Basic  steps  of  the  activity  design  process 

The  design  research  methodology  supported  the  use  of  ethnographic  practices:  qualitative  data,  collected 
from  (2)  Focus  Group  (FG)  sessions  aimed  at  revealing  the  connection  between  what  and  why,  uncovering 
concepts,  relationships  and  processes  while  a questionnaire  has  been  used  for  evaluation  of  both  LAMS 
designed  learning  activities  and  FG  sessions.  Both  activities’  paper  artifacts  and  LAMS  learning  sequence 
archetypes  have  been  used  as  boundary  objects  in  further  development  and  filtering  of  the  learning  activities. 
Generic  templates  of  learning  sequences  and  pedagogical  tools  further  expandable  have  been  developed  so  as 
to  facilitate  learning  design  in  HOU.  The  design  criterion  of  the  activity  template  has  been  to  use  a generic 
form  so  as  :a)  to  motivate  lecturers  to  contribute  to  production  of  new  material  and  increase  participation,  b) 
to  support  and  encourage  the  development  of  learning  examples,  and  emerging  conceptual  models.  Basic 
aim  has  been  to  encapsulate  critical  pedagogical  properties  of  the  design:  theories  of  learning  have  been 
reflected  to  designed  activities.  Grouping  Learning  Design  resources’  aspects  by  discreet  discipline  domains 
is  expected  to  allow  for  simultaneous  investigation  on  what  works  within  domain  using  specific  frameworks 
and  explore  the  relationships  between  the  domains. 


234 


lADIS  International  Conference  e-Learning  2013 


3.2  Preliminary  Data-First  Generation  Pedagogical  Design 

Basic  phases  of  the  User  Oriented  Design  Methodology  are  i)  limiting  the  design  problem  by  analyzing  the 
specific  problem  to  solve,  ii)  building  a theoretical  support  by  building  construction  principles  and  design 
rules,  iii)  building  archetypes  by  the  use  of  boundary  objects  as  a result  of  interaction  between  HOU 
practitioners  and  instructional  designer,  iv)  experimentation  by  two  complementary  phases  and  forthcoming 
scenario  based  evaluation  of  the  boundary  objects  produced,  v)  the  adoption-adaptation  process  which 
progressively  spread  and  transform  the  organizational  context.  These  are  presented  in  the  following  figure; 


1.  Activity  Design 
problem 

,^il 

2.  Building  a 1 

theoretical  support  R 

1 5.  Adjustment  process 

^1 

4.  Experimentation 

3.  Building  Archetypes 
(BOs) 


Figure  2.  Basic  phases  of  the  UO  Methodology 

Boundary  objects  result  from  compromises  and  negotiations  between  designers  and  users,  they  allow  the 
work  of  mutual  adaptation  i.e  the  co  evolution  of  designers’  and  users’  world.  An  important  aspect  stressed 
by  most  authors  is  the  careful  planning  of  all  ODL  activities  from  a pedagogical,  organizational,  financial, 
managerial  and  administrative  point  of  view  (Conole,  2010).  The  types  of  Boundary  Objects  that  have  been 
used  are  presented  in  Figure  3: 


Figure  3.  Types  of  Boundary  Objects  used 
As  main  objectives  of  the  proposed  methodology  have  been: 

• Classification  and  reuse  of  ODL  concepts  and  content  in  HOU  educational  curriculum 

• Integration  and  authorized  exchange/dissemination  of  information  and  knowledge 

• User  friendly  design  i.e.  usage  requires  only  basic  IT  skills 

• Extendibility  and  flexibility  of  educational  content  used 

• Design  and  implementation  of  learning  modules  using  existing  educational  content  and  concepts 
without  excluding  the  emergence  of  new  educational  material 

• Creating  learning  chunks  for  using  LOs 

• Based  on  open  standards  and  existing  LMS  technologies 

Object  Based  Learning  (OBL)  has  been  selected  as  compatible  with  HE  standards  and  amiable  to  already 
produced  educational  content  and  already  designed  Learning  Objects  of  HOU  courses.  Hierarchical  structure 
of  major  and  minor  learning  objectives  in  learning  activities  supports  a)  structuring  of  learning  experience 
from  simpler  to  more  complex  forms,  b)  emergence  of  important  and  less  important  concepts  of  educational 
content  and  principles  in  design  activities,  c)  relatively  easy  transition  from  Micro  level  (activity)  to  Meso 
level  (module)  and  Macro  level  (Thematic  Unit).  Using  reusable  concepts  and  educational  content  of  various 
Thematic  Units  forms  a modular  curriculum  under  which  theory  and  practice  are  closely  interwined  and  less 
important  aspects  of  educational  content  are  left  behind,  providing  space  to  innovation  of  educational  content 
used.  The  design  strategies  developed  in  the  scope  of  the  framework  that  has  been  described  earlier  have 
been  the  following; 
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• Define  learning  outcomes  in  Micro-Meso-Macro  level 

• Structure  Activities-Units-Modules  according  to  learning  outcomes  set  in  specific  criteria  by  HOU 
Lecturers 

• Use  ODL  Standards  to  design  learning  activities,  supported  by  generic  but  flexible  instructional 
design  tools 

• Set  Expected  Learning  Outcomes  but  also  allow  for  emergence  of  non  Expected  Learning  Outcomes 

• Allign  LOs  with  Assessment  tasks  and  use  Learning  Outcomes  as  a springboard  for  emerging 
Learning  Objects 

• Use  both  students  and  teacher  profile  for  the  learning  and  teaching  process:  design  for  diversity 

• Set  criteria  for  learner  performance:  specify  how  students  achieve  knowledge 

• Encourage  deep  level  learning:  from  description  to  explanation 

• Encourage  learner’s  selection:  choosing  between  focus  of  learning  experience 

• Provide  educational  material  which  gradually  becomes  more  difficult  for  the  learner  to  tackle  with 

• Allowing  for  emergence  of  policy  changes  in  organizational  settings 

• Eocus  on  both  concepts  and  skills 

Textual  and  content  mapping  representations  are  probably  the  most  common  ways  in  which  teachers 
think  about  their  designs  (Conole,  2010).  However,  in  this  case  hierarchical  semantic  organization  tools  have 
been  used  in  the  format  of  concept  maps  (expandable  however  in  ontology  schemata)  and  pedagogical 
principles  generated  in  the  (4)  learning  activities.  Pedagogical  tools  initially  designed  and  developed  have 
been  pilot  tested  in  HOU  Practitioners,  process  which  actually  confirmed  the  initial  structure  of  these.  In  a 
design  research  scope  selected  (Eischer  et  al,  2011),  there  has  been  a process  of  reshaping  parameters  of  the 
pedagogical  tools  used,  extending  their  use  in  learning  theories  such  as  self  regulated  learning,  cooperative 
and  constructivist  learning.  The  design  components,  influencing  the  first  generation  pedagogical  design 
which  emerged  out  of  the  implementation  of  our  methodology  are  presented  in  the  Table  1.  Table  2 presents 
produced  combinations  among  Lams  Learning  Activities,  Learning  Outcomes  and  Learning  Object  Types: 


Table  1.  Design  components  influencing  first  generation  pedagogical  design 


Components  Influencing 
Pedagogical  Design 

Category  I 

Category  II 

Learning  Outcomes 

Bloom’s  taxonomy 

Types  of  LOs 

Learner  Characteristics 

Prior  Knowledge 

Learner’s  Profile 

Structure 

Activity  components 

Conceptual  Map 

Information  Presentation 

Activity  structure 

Tasks 

Learning  Environment 

LAMS  tools 

Support  & 

Scaffolding, 

Assessment 

Table  2.  Produced  Combinations  between  LAMS  Learning  Activities,  Learning  Outcomes  and  Learning  Object  Types 


LAMS  Learning  Activity  Learning  Outcomes  Learning  Object 

Types 


Message  Board  Activity 

LOl,  L02,  L03,  L04, 
L05,L08,L09,  LO10,LO13, 

Text 

Share  Resources 

LOS 

Video 

Report  Submission 

LOS 

Exercise 

Questions  and  Answers 

L0S,L06,L011,L012 

Multiple  Choice 
Question 
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In  Figure  4 a snapshot  of  LAMS  Activity  (1)  and  the  conceptual  map  in  Thematic  Unit  (TU)  ICT31  is 
presented; 


Figure  4.  Snapshot  of  LAMS  Activity  (1)  in  HOU  TU  ICT31and  the  conceptual  map  of  the  educational  content 

As  a result  of  the  proposed  methodology  the  following  data  have  been  produced;  (4)  structures  of  learning 
activities  using  LAMS  presenting  however  a variety  in  designed  Learning  Outcomes  and  frequency  of 
LAMS  tools  used  under  different  pedagogical  underpinning  rationale.  Interaction  of  the  research  team  with 
HOU  lecturers  according  to  specific  research  agenda  revealed  important  issues  regarding  the  quality  of 
educational  material  used  in  actual  HOU  courses  and  HOU  Lecturers’  reflection  on  its  use  in  educational 
practice.  The  paper  prototypes  and  digital  archetypes  have  been  evaluated  and  their  structure  confirmed 
during  the  pilot  study  period.  Formative  evaluation  has  been  conducted  using  questionnaires  on  conducting 
Focus  Groups  and  use  of  designed  LAMS  activities.  These  have  provided  feedback  on  the  design 
effectiveness  of  tools  used  and  allowed  us  reach  the  conclusion  that  the  proposed  research  scheme,  its 
products  and  their  basic  operation  has  been  understood  and  accepted  for  use  by  HOU  Practitioners.  Table  3 
presents  Lecturers’  percentage  on  attitudes  regarding  using  basic  categories  of  LAMS  Activities  (Educational 
Content,  Assessment,  Cooperation,  Feedback)  in  (3)  LAMS  Learning  sequences  that  have  actually  been 
tested  to  HOU  students; 

Table  3.  Lecturers’  attitudes  on  usefulness  of  basic  categories  of  LAMS  tools  used  in  (3)  Activities 


L£C^<E«5(9SJ 


4.  CONCLUSION 

The  objective  of  this  paper  is  to  briefly  present  and  propose  basic  characteristics  of  a user  oriented 
methodology  for  designing  and  developing  e-  learning  activities  using  LAMS.  Our  methodology  provides 
new  insight  into  achieving  customization  of  e -learning  activities  and  pedagogical  tools  in  organizational 
contexts.  This  methodology  is  based  on  the  following  characteristics;  a)  the  utilization  of  artifacts  in  the 
practitioners’  activities  and  educational  practice  as  an  opportunity  for  testing,  enriching  and  shifting  them,  b) 
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an  iterative  cycle  process:  building  intermediary  productions  at  each  loop  allows  to  articulate  the  multiple 
rationalities  existing  in  an  educational  network  and  this  articulation  requires  substantial  labor  of  translation, 
negotiation,  debate,  triangulation  and  simplification,  c)  using  boundary  objects  as  mediators  for  the  co 
evolution  of  both  designers’  world  and  practitioners’  world.  We  have  argued  that  the  construction  of 
successive  boundary  objects  and  their  uses  in  real  situations  allows  for  close  interaction  between  scholars  and 
practitioners,  an  emergent  process  of  arbitrage  in  which  practitioners  and  instructional  designers  engage  with 
one  another  to  produce  together  the  solutions  and  both  theoretical  and  practical  knowledge  in  the  field  of 
management.  Future  work  includes  an  iterative  phase  of  testing  the  revised  LAMS  Learning  Activities  in 
HOU  students,  expansion  of  generic  pedagogical  tools  used  and  further  process  and  analysis  of  data 
collected.  A professional  framework  needs  to  be  put  in  place  for  two  purposes:  a)  to  educate  users  about 
learning  design  based  on  pedagogical  principles,  b)  to  facilitate  adoption  in  alignment  with  pedagogical  goals 
set  at  an  institutional  level.  The  development  of  the  pedagogical  framework  needs  time  and  involvement  of 
participants  from  various  jurisdictions  so  as  to  create  rich  pedagogical  structures  that  can  be  used  as  a starting 
point  for  localized  adaptation. 
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ABSTRACT 

Visualizing  the  structure  and  relationships  in  three  dimensions  (3D)  of  organs  is  a challenge  for  students  of  anatomy.  To 
provide  an  alternative  way  of  learning  anatomy  engaging  multiple  senses,  we  are  developing  a force-feedback  (haptic) 
interface  for  manipulation  of  3D  virtual  organs,  using  design  research  methodology,  with  iterations  of  system 
implementation,  formative  evaluation,  and  cyclic  enhancements.  In  the  present  study,  we  aimed  to  determine  the  user 
acceptance  of  the  haptic  interface  for  exploring  anatomical  structures  and  relationships.  Undergraduate  computing 
(n=10)  and  medical  (n=35)  students  from  the  University  of  Tasmania,  Australia,  who  volunteered  to  try  two  iterations  of 
the  system  (n=18  and  27  in  two  separate  user  tests)  reported  on  anonymous  questionnaire  with  quantitative  and 
qualitative  questions  that  the  system  was  easy  to  use,  useful  for  learning,  and  neither  physically  nor  mentally  stressful. 
We  conclude  that  many  medical  students  would  accept  a haptic  interface  for  manipulating  3D  virtual  organs  as  an  aid  to 
learning  anatomy.  Further  development  of  the  system  will  involve  development  of  learning  and  assessment  modules,  and 
we  plan  to  evaluate  the  system’s  usefulness  in  promoting  learning  of  anatomy. 


KEYWORDS 

Anatomy,  haptic  interface,  technology-enhanced  learning,  higher  education,  medical  education 


1.  INTRODUCTION 

Efforts  to  improve  learning  experiences  with  technology  have  been  actively  pursued  for  decades  (Reiser, 
2001).  Different  teaching  strategies  and  learning  resources  vary  in  effectiveness  for  learners  with  different 
learning  styles  (Murphy  et  al.,  2004,  Dobson,  2009);  thus,  offering  a variety  of  learning  options  is 
pedagogically  sound.  This  article  describes  the  initial  stages  of  development  and  evaluation  of  a system  for 
exploring  three-dimensional  (3D)  anatomical  structure  and  relationships  of  human  organs  with  haptic 
feedback. 

As  students  of  human  biology  may  have  only  limited  time  to  learn  from  real  human  organs  or  realistic  3D 
physical  (eg,  plastic)  models,  such  a system  provides  an  additional  option  for  exploring  3D  anatomical 
structures,  making  use  of  new  technological  opportunities  to  enhance  learning  experiences  through  activating 
multiple  sensory  systems  of  the  learner.  Students  have  reported  difficulty  in  learning  anatomy  from  2D 
resources  such  as  textbook  diagrams  (Yeom,  201 1).  Providing  a 3D  model  with  haptic  feedback  reduces  the 
gap  between  a learning  system  and  reality.  When  information  is  presented  in  a multisensory  format,  it 
improves  learning  and  memory  ability  (Shams  and  Seitz,  2008).  A display  device  and  a controlling  device 
are  the  main  computer  peripherals  required  for  this  system.  This  study  investigated  the  acceptance  of  a 
haptic  system  with  3D  anatomical  models  for  learning  anatomy  by  undergraduate  computing  and  medical 
students  at  the  University  of  Tasmania  (UTAS). 

3D  simulations  with  interaction  in  anatomy,  radiological  imaging,  and  surgery  is  an  active  research  area 
(Behringer  et  al.,  2003,  Chien,  2010,  Dieperink,  2006,  Nicholson  et  al.,  2006,  Sakellariou  et  al.,  2009,  Tan  et 
al.,  2012,  Temkin  et  al.,  2006,  Yushkevich  et  al.,  2006,  Liao  et  al.,  2010).  For  example,  learning  from  a 3D 
computer  model  and  standard  two-dimensional  (2D)  images  of  laryngeal  anatomy  did  not  differ,  but  learners 
preferred  the  3D  model  (Tan  et  al.,  2012).  In  contrast,  Nicholson  et  al.  (2006)  found  learning  outcomes  were 
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improved  with  3D  computer  models  of  the  ear  compared  to  traditional  2D  images,  possibly  due  to  greater 
interactivity  and  complexity  of  the  3D  computer  models  vs  the  2D  images.  Computer-generated  3D  models 
may  be  particularly  effective  in  providing  visualization  of  anatomy  that  is  hard  to  represent  through 
traditional  2D  images  (Sakellariou  et  ah,  2009).  User  responses  to  virtual  reality  (VR)  with  a haptic  interface 
were  “positive  and  the  majority  of  users  found  the  VR  system  more  engaging,  interesting  and  easy  to  use  and 
more  efficient  in  elucidating  spatial  inter-relationships  of  structures”  (Sakellariou  et  ah,  2009).  However  it  is 
unclear  whether  the  positive  results  were  due  to  3D  models  produced  in  VR,  the  haptic  interface,  or  a 
combination  of  both.  The  effectiveness  of  a haptic  interface  may  not  be  easy  to  measure  due  to  the 
complexity  of  the  system,  with  interactivity,  quality  of  the  system,  and  learning  outcome  goals  important 
additional  variables  (San  Diego  et  ah,  2012). 

As  educational  research  incorporating  haptic  technology  for  learning  human  anatomy  in  health  fields  is 
currently  mostly  limited  to  advanced,  post-graduate  trainees,  our  research  aims  to  determine  if  a haptic 
interface  can  assist  undergraduate  medical  students  in  learning  anatomy.  Our  overarching  research  question 
is  to  determine  whether  learning  anatomy  in  a 3D  system  with  a force-feedback  palpable  (haptic)  control  will 
enhance  anatomy  learning  and  transfer  of  knowledge  into  realistic  settings.  As  a work-in-progress,  research 
sub-questions  are  to  investigate  if  the  haptic  interface  will  lead  to  more  effective  learning,  whether  users 
accept  a haptic  interface  to  learn  anatomy,  and  what  factors  affect  learning  and  user  acceptance.  This  paper 
focuses  on  evaluating  user  acceptance  of  the  haptic  interface. 


2.  METHODOLOGY 

We  used  the  design  research  approach,  with  iterations  of  system  implementation,  formative  evaluation  and 
cyclic  enhancements  to  develop  our  haptic  system  (Bannan-Ritland,  2003;  Collins  et  ah,  2004;  McKenney, 
2001  and  Reeves,  2006).  To  date,  the  system  has  been  through  two  phases  of  development  and  user  tests. 

2.1  Interface  and  Interaction 

The  interface  domain  in  this  study  comprises  3D  modeling  and  3D  interface  space  with  the  Phantom  Omni 
haptic  device  (Figure  1),  which  provides  six  degrees  of  freedom:  three  for  position  (x,y,z)  and  one  each  for 
pitch,  yaw  and  roll  (rotation  in  the  forward  vertical,  horizontal  and  transverse  planes).  Force  feedback  gives 
different  amounts  of  resistance  to  an  input  depending  on  the  state  of  the  virtual  operation. 


Figure  1 . Haptic  device  (Phantom  Omni)  used  in  the  system  and  user  tests 

The  system  supports  functions  such  as  moving  selected  organs,  zooming,  and  rotating.  Particular  parts  of 
an  organ  can  be  selected  to  display  its  name  (Figure  2),  and  these  labels  can  be  turned  on  or  off. 
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Figure  2.  Screenshot  of  organs  displayed  in  user  test  2.  Several  labels  have  been  turned  on 

The  user  receives  tactile  feedback,  the  force  of  which  depends  upon  the  part  pressed.  For  example, 
different  levels  of  surface  deformation  were  set:  the  lung  is  ‘spongy’  (Figure  3),  while  the  liver  is  firm.  In 
Figure  3,  the  lung  surface  is  being  deformed,  and  the  user  is  experiencing  haptic  feedback. 


Figure  3.  Screenshot  of  the  lungs  displayed  with  surface  deformation  in  user  test  1 

The  organs  displayed  in  user  test  1 included  the  lung,  liver,  and  heart.  For  user  test  two,  the  organs 
displayed  were  the  heart  with  its  four  chambers,  and  numerous  blood  vessels. 

2.2  Development  Environment 

The  computer  system  used  in  user  test  1 was  an  Intel  ® Core  Duo  CPU  @ 2GHz,  RAM  2GB.  An 
upgraded  system  was  used  in  user  test  2,  which  provided  more  sensitive  feedback  to  the  user.  Its  specification 
was  Intel®  Core  ™ 17-2600  CPU  @ 3.4GHz,  RAM  16GB. 

2.3  Formative  Evaluation 

To  investigate  user  acceptance  of  the  haptic  interface,  we  recruited  first  to  third  year  undergraduate  medical 
students  and  computing  students  at  UTAS  to  try  the  system.  For  user  test  1,  an  email  invitation  was  sent  to  all 
(n=230)  second  and  third  year  medical  students.  In  addition,  computing  students  engaged  in  activities  in  a 
computer  lab  were  invited  to  test  the  system,  which  was  set  up  in  an  office.  For  user  test  2,  all  first  year 
medical  students  (n=l  15)  were  invited  to  test  the  system  during  a practical  laboratory  class. 
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Demographic  information  and  prior  experience  with  computers  and  haptic  devices  was  gathered  from 
participants  prior  to  each  test  via  a short  questionnaire.  After  a brief  introduction  to  using  the  system,  each 
user  was  free  to  explore  it.  The  lead  researcher  and  a research  assistant  observed  the  users,  to  observe  how 
they  learned  to  use  the  interface  and  any  difficulties  they  encountered.  After  exploring  the  system, 
participants  were  asked  to  complete  a questionnaire  with  both  visual  analogue  scale  (VAS)  questions  and  an 
open  response  comment  box  on  their  experiences  with  the  system. 

Results  of  quantitative  questions  are  presented  as  mean,  standard  deviation  (SD),  and  standard  error  of  the 
mean  (SEM).  T-tests  were  used  to  compare  results  from  user  tests  1 and  2,  with  significance  level 
conservatively  adjusted  to  a = 0.01  using  the  Bonferroni  correction  for  multiple  comparisons  (type  I family- 
wise  error  rate  of  d = 0.05  divided  by  5,  the  number  of  comparisons).  Quantitative  analysis  was  performed 
in  Excel  and  SPSS.  Qualitative  analysis  of  open  response  comments  was  through  coding  and  identifying 
themes  (Braun  and  Clarke,  2006). 


3.  RESULTS 

3.1  User  Test  1 Settings  and  Demographics  of  Participants 

We  emailed  all  year  2 and  3 Bachelor  of  Medicine,  Bachelor  of  Surgery  (MBBS)  students  at  UTAS  (n=230), 
inviting  them  to  participate  in  our  research  project  by  trying  the  haptic  system  and  completing  a 
questionnaire.  The  system  was  set  up  in  a vacant  tutorial  room  at  the  School  of  Medicine  campus,  three 
times  for  two  hours  each,  when  at  least  some  of  the  students  did  not  have  classes.  In  addition,  some 
computing  students  at  UTAS  were  invited  to  participate  (n=20).  The  system  was  set  up  in  an  office  at  the 
School  of  Computing  and  Information  Systems  for  two  days. 

In  total,  18  students,  15  males  and  3 females,  tried  the  system  and  completed  the  questionnaire,  including 
eight  medical  students  (3.5%  response  rate)  and  ten  computing  students  (50%  response  rate).  The  average 
age  of  all  participants  was  23,  and  four  reported  previous  experience  with  haptic  devices,  but  none  with  the 
Phantom  Omni  haptic  device. 

3.2  User  Test  2 Setting  and  Demographics  of  Participants 

Sample  size  calculations  (n  = 2*[(z„-Zp)CT/(pi-p2)]^),  based  on  an  estimate  of  standard  deviation  (ct)  of  22 
derived  from  the  average  of  the  observed  standard  deviations  on  the  five  VAS  questions  relating  to  user 
acceptance  in  the  first  user  test,  and  a desire  to  detect  a difference  in  mean  scores  (prP2)  of  20  on  the 
100-point  VAS  with  P=0.20  (power  = 0.80,  Zp=-0.84),  yielded  required  sample  sizes  of  n=19  at  a=0.05 
(Zd=1.96)  and  n=29  at  a=0.01  (z„=2.575).  We  emailed  all  year  1 MBBS  students  at  UTAS  (n=115),  inviting 
them  to  participate  in  our  research  project  by  trying  the  system  and  completing  a questionnaire.  The  system 
was  set  up  at  two  practical  laboratory  sessions,  each  two  hours  long  with  half  the  class  each.  Twenty-seven 
students  (23%  response  rate),  16  males  and  11  females,  tried  the  system,  but  only  25  participants  completed 
the  questionnaire,  with  two  participants  each  not  answering  one  question.  More  students  wanted  to  try  the 
system  than  were  able  to,  and  many  students  watched  other  students  using  the  system.  The  average  age  of  all 
participants  was  19,  and  one  reported  previous  experience  with  haptic  devices,  described  as  a mobile  phone. 

3.3  User  Acceptance  of  the  Haptic  System 

Participants  were  asked  to  complete  a questionnaire  immediately  after  using  the  system,  which  included  VAS 
responses  (scale  of  0 to  100)  and  an  open  response  comment  box.  The  design  of  the  questionnaire  was  based 
on  the  Technology  Acceptance  Model  (TAM).  According  to  the  model,  acceptability  of  a system  is 
determined  by  two  main  factors,  perceived  usefulness  and  perceived  ease  of  use  (Chuttur,  2009).  On  average, 
participants  rated  the  interface  useful,  easy  to  use,  indicated  that  they  performed  well  with  the  interface,  and 
reported  low  levels  of  mental  and  physical  stress  (Table  1). 
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Table  1.  Comparison  of  questionnaire  responses  in  user  tests  1 and  2 


Question 

User 

Test 

n 

Mean 

SD 

SEM 

t 

P 

Did  you  perform  well  with 
the  interface?  (0=very  poor 
to  100=very  good) 

1 

18 

71 

19 

4.5 

1.02 

.32 

2 

24 

63 

22 

4.5 

Was  the  interface  useful? 
(0=totally  useless  to 
100=very  useful 

1 

18 

81 

15 

3.5 

1.87 

.07 

2 

24 

69 

17 

3.5 

Was  it  easy  to  use?  (0=very 
difficult  to  100=very  easy) 

1 

18 

73 

18 

4.2 

2.13 

.04 

2 

25 

58 

20 

4.0 

Did  you  become  mentally 
stressed?  (0=not  at  all  to 
100=very  stressed) 

1 

18 

22 

30 

7.1 

-.44 

.66 

2 

25 

32 

27 

5.4 

Did  you  become  physically 
stressed?  (0=not  at  all  to 
100=very  stressed) 

1 

18 

21 

29 

6.8 

1.22 

.23 

2 

25 

17 

19 

3.8 

Observing  the  participants  using  the  haptic  device  during  the  first  user  test,  we  noted  that  when  the  cursor 
was  moved  too  far  in  the  z-axis,  users  found  it  hard  to  bring  the  cursor  back.  This  was  mitigated  by  the  faster 
computer  in  the  second  test.  The  status  of  developed  prototype  was  the  main  difference  between  two  user 
tests.  The  user  test  1 version  included  only  three  organs  (heart,  lungs,  liver),  and  provided  basic  interaction 
with  3D  models  and  haptic  feedback.  The  user  test  2 version  had  28  objects  (heart  chambers  and  blood 
vessels),  and  increased  functionality  including  clicking  to  make  labels  appear.  There  were  some  speed 
issues  with  version  2 when  the  users  tried  to  do  more  complex  activities  such  as  moving  smaller  objects. 

The  questionnaire  responses  show  that  ease  of  use  and  perceived  usefulness  of  the  haptic  interface 
decreased  somewhat  in  the  second  test  compared  to  the  first,  but  these  were  not  statistically  significant  at  p = 
0.01  (adjusted  for  multiple  comparisons).  In  both  user  tests,  most  participants  were  using  the  haptic  interface 
well  after  a few  minutes.  Questionnaire  responses  from  the  second  user  test  indicated  most  participants 
would  like  to  use  the  system  as  an  aid  to  learning  when  fully  developed,  with  an  average  value  of  75  out  of 
100  (n=21  respondents  to  this  question,  SD=17,  SEM=3.8;  0=not  at  all  to  100=very  likely).  Responses  to 
this  question  correlated  positively  with  the  questions  on  usefulness  (r^=0.56),  ease  of  use  (r^=0.31),  and 
performing  well  (r^=0.14);  however,  there  was  no  correlation  with  physical  stress  (r^<0.01),  and  a weak 
negative  correlation  with  mental  stress  (r^=0.07). 

Qualitative  analysis  of  written  comments  on  any  aspect  of  the  system  involved  coding  the  responses 
across  both  user  tests,  with  19  participants  providing  written  comments.  Seven  participants  requested 
additional  functionality,  relating  to  greater  anatomical  detail,  more  options  to  manipulate  organs,  or 
additional  physiology  or  pathology  content.  Seven  participants  described  system  interface  problems,  relating 
to  difficulty  with  placing  the  cursor,  slow  or  jittery  response,  or  the  cursor  getting  stuck.  In  contrast,  five 
participants  praised  the  haptic  feedback  or  other  functionality,  such  as  the  ability  to  rotate  organs.  For 
example,  one  wrote:  “I  felt  that  the  interface  was  appropriate  for  the  task,  with  a high  level  of  fine  control 
permitted.  I was  impressed  with  the  texture  and  density  feedback  test,  there  was  a definite  difference  between 
the  lung  and  liver  in  terms  of  feedback  through  the  haptic  interface.”  One  participant  wrote  they  didn’t  learn 
anything,  another  questioned  usefulness  for  learning,  and  one  described  it  as  “fun”.  In  contrast,  two 
participants  described  the  system  as  a useful  learning  tool,  and  two  others  wrote  they  would  like  to  try  it 
again  when  further  developed. 
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4.  DISCUSSION 

Our  main  finding  was  that  the  majority  of  undergraduate  computing  and  medical  students  who  participated  in 
this  research  study  were  favorable  towards  a haptic  interface  for  exploring  the  structure  and  relationships  of 
virtual  3D  organs  (Table  1).  Our  observations  of  the  participants  during  the  user  tests  suggest  that  students 
can  rapidly  learn  to  use  the  features  of  the  system  (moving  organs,  rotation,  zooming,  pressing  organs).  This 
was  confirmed  in  questionnaire  responses  that  most  users  found  the  system  easy  to  use.  This  concurs  with 
earlier  work  showing  the  ‘naturalness’  of  the  haptic  interface,  and  the  ease  with  which  users  adapt  to  it 
(Massie  and  Salisbury,  1994,  Sankaranarayanan  et  al.,  2003).  The  intuitiveness  and  ease  of  using  a system 
like  this  should  minimize  extraneous  cognitive  load  while  learning  from  the  system.  However,  open  response 
feedback  from  the  questionnaire  indicates  that  further  work  is  needed  to  smooth  responsiveness  of  the  cursor 
to  movement  of  the  haptic  device;  greater  processor  speed  would  assist.  Many  students  would  also  like 
additional  functionality  to  increase  the  potential  to  learn  from  the  system.  Most  participants  indicated  they 
would  like  to  use  the  system  as  an  aid  to  learning  when  it  is  fully  developed,  suggesting  intention  to  use  this 
system.  In  agreement  with  the  technology  acceptance  model  (Chuttur,  2009),  behavioural  intention  to  use 
the  system  correlated  strongly  with  pereeived  usefulness  (‘was  the  interfaee  useful?’,  r^=0.56)  and  perceived 
ease  of  use  (‘was  it  easy  to  use?’,  r^=0.31). 

The  majority  of  participants  in  this  study  were  young  males.  There  were  no  obvious  gender  differences 
in  accepting  the  new  interface,  with  both  males  and  females  expressing  enthusiasm.  The  smaller  number  of 
female  volunteers  may  have  been  because  fewer  females  were  interested  in  trying  a new  technology  system, 
or,  in  the  second  user  test  conducted  during  a regularly  scheduled  class  (laboratory  practical  session),  females 
may  have  been  more  focused  on  their  learning  tasks  than  males.  We  did  not  have  a large  enough  number  of 
participants  to  enable  comparing  responses  by  participant  age.  Anecdotally,  one  mature-age  student  showed 
frustration  and  stopped  using  the  system.  The  current  generation  of  18-20  year  old  undergraduate  students 
are  ‘digital  natives’  who  have  used  technology  since  they  were  5 or  6 years  old  (Prensky,  2001),  confirmed  in 
our  questionnaire  demographic  data.  In  contrast,  some  older  students  may  be  less  comfortable  with  new 
technology. 

The  greater  interest  (response  rate)  in  the  second  test  compared  to  the  first  may  have  been  due  to  students 
seeing  the  haptic  device,  and  watching  others  trying  the  device  and  expressing  enthusiasm.  In  addition,  the 
anatomical  structures  displayed  (heart  and  blood  vessels.  Figure  2)  were  related  to  the  learning  objectives  of 
the  laboratory  practical  session  (relating  to  introductory  anatomy  of  several  organ  systems,  including  the 
cardiovascular  system)  at  which  the  system  was  set  up. 

A limitation  of  this  study  is  potential  self-selection  bias  in  the  samples  of  students  that  tested  the  system. 
In  the  first  user  test,  computing  students  and  year  2 and  3 medical  students  were  invited  to  try  the  system  via 
email;  there  was  a very  low  response  rate  (3.5%)  from  the  medical  students.  In  the  second  test,  the  system 
was  set  up  in  a laboratory  practical  class  for  first  year  medical  students,  and  a higher,  but  still  low  response 
rate  (23%)  was  obtained.  In  both  cases,  the  students  electing  to  participate  in  the  research  and  try  the  system 
may  have  had  a positive  bias  towards  accepting  new  technology.  Thus,  the  high  level  of  user  acceptance,  and 
low  self-reported  levels  of  stress,  may  differ  from  the  general  undergraduate  student  population. 

Although  the  user  acceptance  was  not  statistically  significantly  different  between  the  first  and  second  user 
tests,  there  were  trends  for  decreased  acceptance  levels.  Sample  size  calculations  at  power  of  0.80  (80% 
chance  of  detecting  an  actual  difference)  yielded  n = 19  or  29  at  a=0.05  or  0.01,  respectively.  Our  actual 
numbers  of  respondents  to  the  questions  were  18  and  24-25  in  user  tests  1 and  2,  respectively,  indicating  we 
had  somewhat  less  chance  of  detecting  an  actual  difference  between  our  two  user  tests.  The  trends  towards 
decreased  acceptance  may  have  been  due  to  different  sample  populations.  In  the  first  test,  computing 
students  and  second  and  third  year  medical  students  were  invited  to  participate.  In  the  second  test,  we 
selected  the  group  of  students  more  likely  to  use  the  system  in  its  present  level  of  anatomical  complexity,  that 
is,  first  year  medical  students.  It  is  possible  that  computing  students,  in  particular,  may  have  had  a higher 
level  of  acceptance  of  the  new  technology. 
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5.  CONCLUSION 

We  conclude  that  undergraduate  medical  students  accepted  a haptic  interface  to  explore  the  structure  and 
anatomical  relationships  of  virtual  3D  organs.  However,  the  low  response  rate  and  positive  self-selection 
may  have  resulted  in  a more  positive  response  than  the  population.  Additionally,  positive  responses  may 
have  been  due  to  a novelty  effect  or  that  they  were  being  observed  trying  the  system.  Further  development  of 
our  system  will  provide  another  learning  resource  that  engages  multiple  sensory  systems  for  use  by 
undergraduate  medical  students.  However,  a factor  limiting  wider  availability  of  this  system  is  the 
requirement  for  the  computer  peripheral,  the  haptic  device.  The  next  phase  of  the  research  is  to  develop 
learning  activities  including  formative  assessment  opportunities,  and  testing  materials  to  measure  the  users’ 
learning  achievement  with  the  system  compared  to  traditional  learning  resources. 
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ABSTRACT 

In  the  last  decades  there  have  been  several  attempts  to  use  computers  in  Music  Education.  New  pedagogical  trends 
encourage  incorporating  technology  tools  in  the  process  of  learning  music.  Between  them,  those  systems  based  on 
Artificial  Intelligence  are  the  most  promising  ones,  as  they  can  derive  new  information  from  the  inputs  and  visualize 
them  in  several  meaningful  ways.  This  paper  presents  an  application  of  machine  learning  to  music  performance  which  is 
able  to  discover  the  similarities  and  differences  between  a given  performance  and  those  from  other  musicians.  Such  a 
system  would  help  students  to  better  learning  how  to  perform  a certain  piece  of  music,  allowing  them  to  compare  with 
other  students  or  master  performers. 
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1.  INTRODUCTION 

Student  musicians  spend  most  of  their  time  practising.  These  countless  hours  of  practising  help  them  learn  to 
interpret  a piece  of  music  as  the  composer  imagine  it,  but  also  developing  their  own  style  -one  that  is  unique 
to  each  of  them.  Said  in  other  words,  what  makes  a piece  of  music  come  alive  is  also  what  distinguishes  great 
artists  from  any  other. 

In  this  work,  we  propose  a piece  of  software  and  a machine  learning  algorithm  which  are  able  to  classify 
performances  according  to  their  similarity  to  those  of  famous  musicians.  Such  a system  would  help  students 
to  understand  the  musical  resources  some  of  the  greatest  performers  use  and  how  to  imitate  them. 

We  focus  on  the  task  of  characterizing  performers  from  their  playing  style  using  descriptors  that  are 
automatically  extracted  from  commercial  audio  recordings  by  means  of  state-of-the-art  feature  extraction 
tools.  This  learning  process  is  done  by  employing  the  system  described  in  Molina-Solana  et  al.  (Molina- 
Solana,  M.  et  ah,  2010).  This  approach  is  quite  different  from  those  in  the  literature,  in  which  a heavy  human 
intervention  is  needed  in  order  to  manually  annotate  the  music.  That  is,  however,  what  makes  it  feasible  for 
being  employed  in  an  automated  system  such  as  the  one  we  present  here.  The  devised  software  requires  no 
human  intervention,  being  the  whole  process  done  in  an  automatic  way. 

Software  tools  aimed  to  help  students  to  better  perform  have  mainly  the  drawback  that  they  are  limited  to 
a range  of  previously  analysed  songs.  That  is  not  the  case  of  our  system  as  far  as  the  learning  model  we  have 
chosen,  is  able  to  capture  how  the  performer  plays,  regardless  the  piece  being  performed.  This  way,  the 
student  is  assessed  of  how  to  play  a work  even  though  the  expert  musician  did  never  play  it. 

No  special  hardware  (e.g.  MIDI  instruments)  is  required  to  appraise  the  student’s  unique  way  of 
performing.  Only  an  audio  recording  of  the  student  performing  with  their  own  instrument  is  required.  Thus, 
the  performance  is  no  biased  in  any  sense:  students  play  with  their  own  instrument  and  there  is  no  need  of 
them  being  aware  of  them  being  recorded.  In  fact,  it  is  relatively  easy  to  count  with  students’  recordings 
nowadays  because  many  teachers  encourage  their  students  to  record  themselves  when  playing. 

This  paper  is  organized  as  follows.  Firstly,  in  Section  2 we  provide  some  background  about  both  Music 
Performance  and  Music  Education.  In  section  3,  we  briefly  explain  the  model  we  use  to  represent  the 
information  about  performances,  and  how  that  information  is  extracted  from  audio  files.  Section  4 presents 
the  devised  software  and  how  it  can  be  used.  Finally,  Section  5 points  out  some  conclusions  and  further 
developments. 
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2.  BACKGROUND 

This  section  describes  what  Music  Performance  is,  and  how  computers  can  be  used  for  Music  Education. 
Former  works  in  those  areas  are  also  revised  to  provide  the  reader  a proper  framework  for  the  rest  of  the 
paper. 

2.1  Music  Performance 

The  literal  execution  of  a musical  score  is  frequently  judge  as  being  significantly  less  interesting  than  a 
performance  of  that  piece  by  even  a moderately  skilled  musician.  The  reason  is  straightforward:  musicians  do 
not  produce  literal  renditions  of  the  score.  That  is,  they  do  not  play  a piece  of  music  mechanically,  with 
constant  tempo  or  loudness,  exactly  as  written  in  the  printed  music  score.  Rather,  skilled  performers  slow 
down  at  some  places,  speed  up  at  others  and  stress  certain  notes  or  passages.  Tempo  variations  and  loudness 
variations  are  the  most  important  parameters  available  to  a performer,  being  also  the  main  source  of 
expression  in  music  (Palmer,  C.,  1996).  The  way  these  parameters  ‘should  be’  varied  is  not  precisely 
specified  in  the  printed  score;  so  that,  it  is  the  performer  the  one  in  charge  of  using  them  properly. 

According  to  Widmer  and  Goebl  (Widmer,  G.  and  Goebl,  W.,  2004),  expressive  music  performance  can 
be  defined  as  the  deliberate  shaping  of  the  music  by  the  performer,  in  the  moment  of  playing,  by  means  of 
continuous  variations  of  parameters  such  as  timing,  loudness  or  articulation.  Changes  in  tempo  (timing)  are 
modifications  of  the  regular  grid  of  beats  that  defines  time  in  a score.  Changes  in  loudness  (dynamics)  are 
modifications  of  the  intensity  of  notes  with  respect  to  the  others  and  to  the  general  energy  of  the  fragment  in 
consideration.  Articulation  consists  in  varying  the  gap  between  contiguous  notes  by,  for  instance,  making  the 
first  one  shorter  or  overlapping  it  with  the  next. 

Music  performance  is  a complex  activity  involving  physical,  acoustic,  psychological,  social  and  artistic 
issues.  At  the  same  time,  it  is  also  a deeply  human  activity,  relating  to  emotional  as  well  as  cognitive  and 
artistic  categories.  Furthermore,  it  is  dramatically  affected  by  performer’s  mood  and  physical  condition. 

As  said  before,  research  in  music  performance  has  a multidisciplinary  character,  with  studies  that  range 
from  understanding  expressive  behaviour  to  modelling  aspects  of  renditions  in  a formal  quantitative  and 
predictive  way.  Historically,  research  in  expressive  music  performance  has  focused  on  finding  general 
principles  underlying  the  types  of  expressive  ‘deviations’  from  the  musical  score  (e.g.,  in  terms  of  timing, 
dynamics  and  phrasing)  that  are  a sign  of  expressive  interpretation.  Works  by  De  Poli  (Poli,  G.  D.,  2004), 
Widmer  and  Goebl  (Widmer,  G.  and  Goebl,  W.,  2004),  and  Delgado  et  al.  (Delgado,  M.  et  ak,  2011)  contain 
overviews  on  expressive  performance  modelling.  The  reader  should  refer  to  them  for  further  information. 

One  of  the  issues  in  this  area  is  the  representation  of  the  way  certain  performers  play  by  just  analysing 
some  of  their  renditions  -studies  into  the  individual  style  of  famous  musicians.  That  information  would 
enable  us  to  identify  performers  by  just  listening  to  their  renditions  (Molina-Solana,  M.  et  ak,  2010; 
Saunders,  C.  et  ak,  2008;  Stamatatos,  E.  and  Widmer,  G.,  2005;  Widmer,  G.  et  ak,  2003).  These  studies  are 
difficult  because  the  same  professional  musician  can  perform  the  same  score  in  very  different  ways.  Among 
the  methods  for  the  recognition  of  music  performers  and  their  style,  the  most  relevant  are  the  fitting  of 
performance  parameters  in  rule-based  performance  models,  and  the  application  of  machine  learning  methods 
for  the  identification  of  performing  style  of  musicians.  Recent  results  of  specialized  experiments  show 
surprising  artist  recognition  rates  (for  instance,  see  those  from  Saunders  et  al.  (Stamatatos,  E.  and  Widmer, 
G.,  2005)  or  Molina-Solana  et  al.  (Molina-Solana,  M.  et  ak,  2010)). 

2.2  Computers  in  Music  Education 

According  to  Brown  (Brown,  A.R.,  2007),  there  are  three  main  roles  in  which  a computer  can  take  part  in 
music  education:  they  can  act  as  a tool,  a medium  or  a musical  instrument.  A tool  for  recording,  editing, 
analyzing  and  sequencing  sounds;  a medium  for  music  storage,  indexing  and  distribution;  and  a musical 
instrument  for  synthesizing  music  in  real  time. 

In  the  last  decades  there  have  been  several  attempts  to  use  computers  in  Music  Education.  The  field  is 
highly  interdisciplinary,  involving  contributions  from  disciplines  such  as  Music,  Education,  Artificial 
Intelligence,  Psychology,  Linguistics,  Human  Computer  Interaction  and  many  others.  Advances  in  research. 
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as  well  as  new  software  tools  for  the  analysis  of  data,  open  up  a new  area  in  the  field  of  music  education,  as 
stated  by  Friberg  and  Battel  (Friberg,  A.  and  Battel,  G.  U.,  2002). 

Artificial  Intelligence  in  Music  Education  is  also  a very  diverse  field,  being  the  work  by  Holland 
(Holland,  S.,  2000)  still  an  interesting  review  in  the  topic.  Attempts  in  this  area  can  be  classified  into  four 
categories,  according  to  Brandao  et  al.  (Brandao  et  ah,  1999),  which  are  intended  to:  teach  fundamentals  of 
music,  teach  musical  composition  skills,  perform  analysis  of  music,  and  teach  musical  performance  skills.  In 
this  paper,  we  propose  an  example  within  the  scope  of  the  last  group. 

However,  music  performance  is  not  a well-defined  domain.  There  are  no  clear  goals,  correct  answers  or 
an  only  way  of  doing  things.  That  is  probably  why  it  is  a tough  domain  researchers  are  not  willing  to  face. 
Persson  et  al.  (Persson,  R.  S.  et  al.,  1996)  pointed  it  out  when  they  stated  that  relatively  little  time  is 
dedicated  to  interpretative  and  emotional  aspects  of  performance  in  comparison  with  that  dedicated  to 
learning  notation  -a  domain  that  can  be  more  easily  represented. 

So,  why  using  technology  for  such  a task?  Despite  computers  will  hardly  replace  music  teachers,  they  can 
complement  them.  The  benefits  of  individualized  instruction,  assessment  and  motivation  can  be  used  to 
supplement  the  learning.  Using  technology,  students  can  work  at  their  own  pace,  focusing  on  those  aspects 
they  need  to  improve.  Because  the  learning  is  individual,  there  is  no  peer  competition  that,  in  many  cases,  is 
counterproductive.  Evaluation  of  responses  can  be  totally  accurate  and  impartial.  These  possibilities  have 
been  tested  with  promising  results  by  Eriberg  and  Battel  (Eriberg,  A.  and  Battel,  G.  U.,  2002). 

New  pedagogical  trends  suggest  recording  students’  performances.  This  way,  they  can  hear  how  they  play 
and  analyse  critically  their  work.  Listening  to  both  professional  and  amateur  recordings  is  also  a valuable 
teaching  opportunity  for  music  students.  On  the  one  hand,  this  allow  them  to  compare  their  own  current 
performance  to  those  of  professional  musicians,  being  a motivational  way  of  helping  students  to  set  goals  for 
practising.  On  the  other  hand,  students  can  identify  improvements  with  respect  to  previous  recordings  and  be 
aware  of  their  progression  and  achievements  during  the  time. 

Beside  the  role  recordings  play  to  identify  errors,  the  use  of  recordings  can  also  be  useful  for  going 
further  and  focus  on  specific  musical  elements  or  sections,  providing  a deeper  understanding  of  the 
performance.  Eurthermore,  it  gives  students  the  possibility  of  record  themselves  in  other  places  different  than 
the  classroom,  releasing  them  of  the  pressure  of  a classroom  situation  and  allowing  them  to  better  perform. 

Once  recordings  are  available,  we  can  also  take  advantage  of  them  by  means  of  computational  tools,  like 
the  one  we  are  presenting  in  this  paper.  These  tools  might  be  able  to  point  out  the  tempo  of  the  work,  to 
discover  irregularities  in  the  rhythm,  or  to  stress  some  musical  parameters.  Of  special  interest  is  the 
possibility  of  representing  the  sound  in  a new  meaningful  way,  like  in  the  works  by  Lagner  and  Goebl 
(Langner,  J.  and  Goebl,  W.,  2003)  or  Saap  (Sapp,  C.,  2007).  In  our  case,  we  are  able  to  offer  a list  of 
performers  that  played  in  a similar  way  of  the  input  recording. 

The  analytical  comparison  between  a natural  performance  and  performances  with  particular  expressive 
intentions  also  seems  to  possess  a potential  for  music  pedagogy.  One  of  these  tools  is  Director  Musices 
(Friberg,  A.  et  al.,  2000)  which  is  able  to  play  a song  applying  some  performance  rules.  This  way,  certain 
acoustic  parameters  can  be  exaggerated  so  that  anybody,  regardless  of  their  musical  training,  can  detect  the 
differences  and  concentrate  on  particular  aspects  of  the  performance. 


3.  LEARNING  ALGORITHM 

Our  approach  for  dealing  with  the  characterization  of  performers  is  based  on  the  acquisition  of  trend  models, 
which  represent  each  particular  performance.  As  said  before,  we  employ  the  learning  algorithm  described  in 
Molina-Solana  et  al.  (Molina-Solana,  M.  et  al.,  2010).  It  is  a quick  and  automatic  method  for  collecting 
information  about  the  performer  of  a musical  piece  from  its  audio.  It  obtains  an  approximate  score  with 
information  about  pitch  and  energy.  Because  the  method  is  automatic,  there  is  no  manual  annotation  and  a 
confident  musical  analysis  is  not  performed. 

Despite  those  apparent  drawbacks,  the  alternative  the  trend  model  presents  consist  in  taking  a more  global 
perspective  of  a performer  by  trying  to  capture  his  essential  (and  recurrent)  expressive  decisions.  This 
method  does  not  focus  in  concrete  cues,  but  in  the  general  tendency.  Because  only  an  approximated  score 
derived  from  the  audio  is  available,  a melodic  contour  segmentation  is  employed  as  a way  to  capture  the 
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musical  structure  of  the  piece.  This  segmentation  allows  an  analysis  in  terms  of  melodic  intervals,  i.e. 
precision  on  pitch  detection  is  not  critical. 

Specifically,  a trend  model  characterizes,  for  a specific  audio  descriptor,  the  relationships  a given 
performer  is  establishing  among  groups  of  neighbour  musical  events.  A qualitative  analysis  of  the  variations 
of  the  audio  descriptors  is  performed  with  a local  perspective.  Two  trend  models  will  be  used:  energy  and 
duration.  The  trend  model  for  the  energy  descriptor  relates,  qualitatively,  the  energy  variation  for  a given  set 
of  consecutive  notes,  and  it  is  related  to  dynamics.  On  the  other  hand,  the  trend  model  for  duration  indicates, 
also  qualitatively,  how  note  durations  change  for  note  sequences.  Duration  is  related  to  articulation  and 
timing. 

Given  an  input  musical  recording  of  a piece,  the  trend  analysis  is  performed  by  aggregating  the  qualitative 
variations  on  their  small  melody  segments.  Thus,  in  advance  of  building  trend  models,  input  streams  are 
broken  down  into  segments  of  three-note  long.  As  most  automatic  melody  segmentation  approaches  do,  note 
grouping  is  performed  according  to  a human  perception  model.  At  the  training  stage,  the  goal  of  our  system 
is  to  characterize  performers  by  extracting  expressive  features  and  constructing  trend  models.  Next,  at  the 
identification  stage,  the  system  analyses  the  input  performance  and  looks  for  the  most  similar  previously 
learned  model. 

3.1  Data  Representation 

As  said,  the  first  step  consists  in  extracting  audio  features.  So  far,  we  only  consider  fundamental  frequency 
and  energy,  as  these  are  the  main  low-level  audio  features  related  to  melody.  These  features  are  then  used  to 
identify  note  boundaries  and  to  generate  melodic  segments.  This  module  provides  a vector  with  instantaneous 
fundamental  frequency  and  energy  values  calculated  every  0.01  seconds. 

After  some  post-processing  for  deleting  noise,  a smooth  vector  of  pitches  is  obtained.  By  knowing  on 
which  samples  pitches  are  changing,  a note-by-note  segmentation  of  the  whole  recording  is  performed.  For 
each  note,  its  pitch,  duration  and  energy  are  collected. 

It  is  assumed  that  there  might  be  some  errors  in  this  automatic  segmentation,  given  the  heterogeneity  of 
recording  conditions.  The  trendmodel  algorithm  proposes  a more  abstract  representation  (but  still  close  to  the 
melody)  than  the  real  notes  for  dealing  with  this  problem.  That  is,  instead  of  focusing  on  the  absolute  notes, 
we  are  interested  in  modelling  the  melodic  surface. 

To  do  that,  a simple  contour  model  that  identifies  some  melodic  patterns  from  the  melody  is  used.  These 
patterns  are  three-note  long  and  are  related  with  the  direction  of  the  two  intervals  that  exist  in  them. 

3.2  Classification 

A nearest  neighbour  classifier  is  used  to  generate  the  list  of  similar  performers  for  each  input  recording. 
Trend  models  acquired  in  the  training  stage,  as  described  in  the  previous  section,  are  used  as  class  patterns. 
When  the  student’s  recording  is  presented  to  the  system,  its  trend  model  is  created  and  compared  with  the 
previously  acquired  ones.  The  system  outputs  a ranked  list  of  performer  candidates  where  distances 
determine  the  order,  with  1 being  the  most  likely  performer  relative  to  the  results  of  the  training  phase. 

The  distance  dij  between  two  performances  i and  j,  is  defined  as  the  weighted  sum  of  distances  between 
respective  contour  patterns.  Weights  have  been  introduced  for  balancing  the  importance  of  the  patterns  with 
respect  to  the  number  of  times  they  appear.  Frequent  patterns  are  considered  more  informative  due  to  the  fact 
that  they  come  from  more  representative  samples.  When  several  audio  descriptors  are  considered,  the 
individual  corresponding  distances  are  aggregated.  Readers  should  refer  to  the  original  paper  describing  this 
algorithm  (Molina-Solana,  M.  et  ak,  2010)  in  order  to  find  more  details  and  the  exact  equations. 

3.3  Model  Performance 

The  feasibility  and  accuracy  of  the  system  have  already  been  tested  by  using  several  works  from  Sonatas  and 
Partitas  by  J.S.  Bach  (Molina-Solana,  M.  et  ak,  2010).  In  those  experiments,  the  model  was  employed  to 
identify  several  violinists  by  using  some  already  labelled  performances.  The  results  were  quite  promising; 
much  better  than  those  from  a random  chance  classifier  and  probably  better  that  those  achievable  by  a human 
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expert.  It  was  also  demonstrated  that  the  information  that  trend  models  contain  can  be  used  to  derive 
meaningful  musicological  data. 

The  system  has  only  been  tested  with  violin  recordings,  but  the  underlying  machine  learning  model 
(trendmodel  algorithm)  is  flexible  enough  to  allow  the  employment  of  other  monophonic  instruments.  The 
application  itself  is  capable  of  dealing  with  any  instrument  or  song  as  long  as  the  information  it  receives  is 
properly  formatted. 


4.  SOFTWARE 

We  have  implemented  in  MATLAB  all  the  tools  for  extracting  the  information  from  musical  signals, 
representing  those  data,  and  using  it  for  comparing:  the  whole  trend-model  algorithm  and  an  interface  to  use 
all  the  developed  tools. 

The  software  is  very  user-friendly.  The  procedure  is  as  follow:  1)  the  user/student  selects  the  file  with 
their  performance;  2)  selects  the  files  to  compare  with;  3)  waits  for  the  audio  analysis  at  the  learning  stage;  4) 
launches  the  comparator. 

The  devised  interface  (see  Figure  1)  enables  the  user  to  load  a set  of  files  to  be  used  as  comparing 
performances.  On  the  left,  we  find  a list  of  all  the  audio  files  with  performances  that  we  have  in  the  folder. 
From  those,  we  can  select  only  those  we  are  interested  in  being  used.  On  the  right,  we  select  the  audio  file 
containng  the  student’s  performance  we  want  to  compare  with  the  others.  The  developed  software  uses  audio 
files  in  wav  format  as  input. 


Learn  Set 

iGarrett  Fiichbach  wav 
Ijaap  Schroder  wav 

[Mela  Tenenbaum.wav 
Nathan  Milstein.wav 


Rachel  Podger.> 


Sergiu  Luca.wav 
Shlomo  Mintz.wav 
Yehudi  Menuhin  .wav 
my  performance  wav 


My  song: 

my  performance.wav 


Browse. . 


Process! 


Show  info 


Distances! 


|—  Attributes  — 

0 Duration 
[7]  Energy 


Compare! 


Figure  1.  Main  window  of  the  devised  interface  of  the  system 

Once  we  have  selected  the  files  we  want  to  use,  the  system  needs  to  extract  information  from  them,  and 
represent  them  by  means  of  trend  models.  That  is  done  when  the  ‘Process’  button  is  pressed.  The  interface 
offers  the  student  the  possibility  to  choose  which  (or  all)  attributes  they  want  to  use  when  comparing 
performances  (‘Attributes’  section  in  the  interface).  Currently,  only  two  possibilities  are  allowed  as  they  are 
the  attributes  used  in  the  learning  stage:  duration  of  notes  and  their  energy.  Those  two  attributes  are  the  main 
ones  in  music  performance,  as  previously  explained  in  Section  2.1.  Thus,  students  can  compare  themselves  in 
terms  of  timing,  in  terms  of  energy,  or  in  both  together.  Results  may  differ  depending  on  the  selection. 
However,  any  other  musical  attribute  can  be  used  as  long  as  they  are  extracted  from  the  audio  and  learned  in 
the  learning  stage. 

‘Distances’  shows  a distance  matrix  comparing  the  distances  between  each  pair  of  performances 
(including  the  one  by  the  student).  An  example  can  be  seen  in  Figure  2.  A 2D-cluster  representation  of  the 
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performances  is  also  shown  (Figure  3).  This  representation  is  done  by  comparing  the  distances  between  each 
pair  of  performances  (in  a high  dimension  space)  and  projecting  them  to  a plain.  For  instance,  here  we  can 
see  that  the  student’s  performance  is  pretty  similar  to  that  of  Jaap  Schroder  (always  according  to  our 
described  method  for  representing  performances),  and  quite  different  from  Garret  Fischbach’s. 


Figure  2.  Normalized  similarity  matrix  between  performances 


[fischbach] 


Cmilstein] 


Cbrooks] 


(tenenbaum  ] 


[MENUHIN] 


[malikian  1 


[fischer] 


[podger] 


[mintz] 


(me]  (SCHRODER) 


(luca] 


Figure  3.  Cluster  representation  of  the  performances 

This  cluster  representation  gives  an  idea  of  how  the  student  plays  in  comparison  with  others.  Flowever,  it 
cannot  tell  where  the  differences  are  nor  the  changes  to  be  made  to  their  style  in  order  get  closer  to  a certain 
performer.  ‘Show  info’  offers  a representation  of  trend  models.  They  are  represented  as  histograms.  Each  bar 
indicates  how  a performer  tends  to  play  for  a certain  musical  pattern.  More  details  can  be  found  in  the  paper 
where  the  algorithm  is  described  (Molina-Solana,  M.  et  al.,  2010). 

Finally,  we  can  obtain  a list  of  sorted  performers  (‘Compare’  button),  with  the  first  one  being  the  most 
similar  to  the  student  one,  and  the  last  being  the  most  different.  The  software  offers  the  result  as  an  image 
(see  Figure  4).  In  black,  we  find  the  most  similar  performer,  whereas  in  white  is  the  furthest  one. 
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Y.  Menuhin 

R.  Rodger 
G.  Fishbach 
B.  Brooks 
A.  Malikian 
J.  Fischer 

S,  Mintz 
N.  Milstein 
J.  Schroder 

M.  Tenenbaum 
S.  Luca 

my  performance 

Figure  4.  Ranking  of  similar  performers 


5.  CONCLUSION 

In  this  paper,  we  have  presented  an  application  of  machine  learning  to  music  performance.  The  software  we 
have  developed  is  able  to  discover  and  show  the  similarities  and  differences  between  a given  performance 
and  those  from  other  musicians.  In  our  opinion,  such  a system  would  be  of  great  interest  for  music  students, 
as  it  can  help  them  to  better  perform  a piece  of  music.  It  shows  them  the  similarities  and  differences  with 
those  renditions  of  famous  performers. 

Technology  applied  to  music  education  is  a very  promising  area.  New  pedagogical  tendencies  suggest 
recording  students’  performances.  Our  system  uses  precisely  those  recordings  to  proceed,  not  needing  any 
other  special  hardware  to  collect  the  information.  Doing  so,  we  get  another  advantage:  all  the  data  is  gathered 
in  a transparent  way,  not  interfering  with  the  performance.  We  have  employed  and  algorithm  already  in  the 
literature  which  matches  well  with  our  constraints. 

Research  on  music  performance  can  point  out  expressive  resources  that  traditionally  have  been  hiding  in 
musicians’  skill  and  musical  intuition.  When  explicitly  formulated,  these  resources  will  give  the  user  the 
possibility  to  play  music  with  different  expressive  colouring.  Even  more,  they  would  allow  a computer  to  do 
so.  Not  in  vain,  the  art  of  performing  music  is  not  well-defined  and  it  is  the  result  of  several  years  of  training; 
a knowledge  that  is  very  difficult  to  be  appraised  by  a computer.  Thus,  studies  in  music  performance  are  of 
great  value  in  our  time. 
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ABSTRACT 

Although  digital  games  have  been  developed  for  various  subject  areas,  little  attention  has  been  focused  on  using  digital 
games  to  address  international  students’  adjustment  issues.  For  this  reason,  this  paper  endeavors  to  explore  the  use  of  a 
digital  game  in  facilitating  international  students  acquire  adjustment-related  information.  Specifically,  the  objectives  of 
this  paper  are  twofold.  One,  it  seeks  to  introduce  a digital  game  called  Digital  Game  for  International  Student  Training 
(DGIST)  which  is  intended  to  satisfy  important  aspects  of  international  students’  information  needs  through  fun.  Two,  it 
seeks  to  perform  a preliminary  evaluation  of  DGIST  in  terms  of  its  efficacy  in  helping  international  students  acquire 
adjustment-related  information  through  a before-and-after  with  control  experimental  design.  A total  of  80  participants 
were  involved  in  the  study.  Despite  a few  shortcomings,  the  results  for  DGIST  appeared  to  be  promising.  Statistical 
analyses  confirmed  that  DGIST  was  more  effective  in  helping  students  acquire  adjustment-related  information  than  the 
paper-based  document.  This  paper  concludes  with  some  limitations  and  suggests  a few  areas  for  further  research. 


KEYWORDS 

Information  need,  adjustment,  fun,  gameplay,  international  student 


1.  INTRODUCTION 

One  of  the  outcomes  of  globalization  today  is  the  growing  number  of  international  students  who  are  keen  to 
gain  knowledge  and  professional  skills  in  different  countries.  Although  studying  in  a cross-cultural  setting 
may  be  valuable,  international  students  may  experience  challenges  and  acculturative  stress  in  adjusting  to  the 
new  environment  (Poyrazli  and  Grahame,  2007).  Consequently,  the  feelings  of  confusion,  anxiety,  sense  of 
loss  and  isolation  may  arise.  If  not  managed  properly,  international  students’  academic  success,  social  well- 
being and  psychological  health  could  be  in  jeopardy  (Hechanova-Alampay  et  ah,  2002). 

Challenges  associated  with  adjustment  faced  by  international  students  could  be  mitigated  using  digital 
games.  Exerting  positive  effects  on  understanding,  learning  and  practicing  new  knowledge,  it  is  no  surprise 
that  digital  games  have  been  increasingly  deployed  for  pedagogical  purposes  (Gee,  2003).  By  infusing  the 
element  of  enjoyment,  the  barrier  to  learning  is  lowered  (Li  et  ah,  2013;  Prensky,  2001).  Although  digital 
games  have  been  developed  for  various  subject  areas  ranging  from  information  systems  (Martin,  2000)  to 
second  language  learning  (Kongmee,  et  ah,  2011),  little  attention,  both  in  terms  of  research  and  practice,  has 
been  focused  on  using  digital  games  to  address  international  students’  adjustment  issues.  For  this  reason,  this 
paper  endeavors  to  explore  the  use  of  a digital  game  in  facilitating  international  students  acquire  adjustment- 
related  information.  Specifically,  the  objectives  of  this  paper  are  twofold.  One,  it  seeks  to  introduce  a digital 
game  called  Digital  Game  for  International  Student  Training  (DGIST)  which  is  intended  to  satisfy  important 
aspects  of  international  students’  information  needs  through  fun.  Two,  it  seeks  to  perform  a preliminary 
evaluation  of  DGIST  in  terms  of  its  efficacy  in  helping  international  students  acquire  adjustment-related 
information.  The  evaluation  was  conducted  using  a before-and-after  with  control  experimental  design. 
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The  rest  of  the  paper  is  structured  as  follows.  Section  2 describes  a literature  review  on  international 
students’  adjustment  issues  and  their  corresponding  information  needs.  In  addition,  the  fun  aspects  to  make  a 
digital  game  enjoyable  are  reviewed.  Section  3 explains  the  design  overview  of  DGIST  which  is  built  on  the 
literature  described  earlier.  The  research  methodology  and  the  evaluation  process  of  the  game  are  presented 
in  section  4.  The  results  of  the  evaluation  are  reported  in  Section  5 while  insights  drawn  from  evaluation  are 
highlighted  in  Section  6.  Finally,  section  7 concludes  the  paper  with  a note  for  future  research  directions. 


2.  LITERATURE  REVIEW 


2.1  International  Students’  Adjustment  Issues 

As  the  demand  for  oversea  education  increases,  issues  related  to  intercontinental  adjustment  come  to  be  fore 
(Tseng  and  Newton,  2002).  Studying  in  a foreign  country  and  experiencing  sharp  changes  in  culture, 
language  and  the  surrounding  environment  may  cause  acculturative  stress  and  adjustment  problems.  Extant 
literature  suggests  at  least  four  categories  of  adjustment  issues,  namely,  preliminary  needs,  cultural, 
educational  and  psychological  challenges. 

When  international  students  move  to  another  country,  they  may  face  some  difficulties  in  satisfying  their 
preliminary  needs,  such  as  finding  a suitable  accommodation,  applying  for  social  security  ID,  learning  how  to 
use  transportation  system,  adapting  to  local  food  and  being  familiar  with  campus.  If  these  basic  information 
needs  remain  unaddressed  upon  arrival,  international  students  will  likely  be  stressed  (Hechanova-Alampay  et 
ak,  2002).  To  deal  with  these  preliminary  adjustment  problems  researchers  suggest  different  methods 
including  a tour  of  the  campus,  orientation  program,  interacting  with  other  senior  overseas  compatriot 
students  (Poyrazli  and  Grahame,  2007)  and  expanding  individual  world  view  (Tseng  and  Newton,  2002). 

Cultural  differences  are  one  of  the  most  significant  adjustment  challenges  for  international  students  (Zhai, 
2004).  In  particular,  unfamiliarity  with  cultural  references,  idioms,  sarcasm,  slang  and  missing  cultural 
specific  cues,  such  as  verbal  and  non-verbal  messages  affect  overseas  students’  tendency  to  seek  out  social 
interaction  in  the  host  culture  (Olivas  and  Li,  2006).  An  effective  way  to  overcome  these  issues  is  to  connect 
to  the  host  people  or  participate  in  their  cultural  activities  (Tseng  and  Newton,  2002).  Creating  social 
interaction  with  local  students  is  another  way  to  foster  cultural  adaption.  In  parallel,  international  students 
who  positively  adjust  should  be  encouraged  to  offer  help  to  new  incomers  (Olivas  and  Li,  2006). 

Although  academic  systems  are  fairly  comparable  all  over  the  world,  they  are  not  completely  identical 
(Ridley,  2004).  Conceivably,  international  students  in  their  first  year  of  study  face  more  educational 
challenges  than  their  local  counter-parts  (Zhao  et  ak,  2005).  Furthermore,  language  barrier  in  the  academic 
environment  can  aggravate  the  situation.  International  students  can  overcome  these  educational  challenges  by 
asking  for  help  from  other  students  or  faculty  (Tseng  and  Newton,  2002).  Also,  the  role  of  orientation 
programs  in  transferring  university  information  should  not  be  ignored. 

International  students  experience  some  inevitable  psychological  challenges  such  as  acculturative  stress, 
homesickness,  loneliness,  frustration,  depression  or  isolation  and  feelings  of  worthlessness  (Olivas  and  Li, 
2006;  Tseng  and  Newton,  2002).  Since  they  are  far  from  their  family  and  friends,  they  experience  less  social 
support  vis-a-vis  local  students  (Hechanova-Alampay  et  ak,  2002).  Communicating  with  other  students  and 
creating  a social  network  may  mitigate  their  need  for  social  support  (Poyrazli  and  Grahame,  2007). 

2.2  Eun  Principles  in  Digital  Games 

Games  are  a subset  of  play  and  fun  (Prensky,  2001).  People  play  games  because  they  are  seeking  fun  (Bartle, 
2004).  As  an  organized  play,  digital  games  not  only  offer  enjoyment  but  could  be  used  to  heighten  the 
motivation  for  learning  and  improve  academic  achievement  (Kebritchia,  et  ak,  2010). 

To  unravel  the  amorphous  concept  of  fun  in  digital  games,  a theoretical  framework  which  comprises  three 
main  factors,  namely,  immersion,  challenges  and  social  interaction  has  been  developed  as  shown  in  Table  1. 
Each  of  these  factors  comprises  overlapping  sub-factors. 

Immersion  refers  to  the  extent  to  which  a player  is  engaged  with  a game  (McMahan,  2003).  Factors  that 
lead  to  immersion  include  deep  involvement,  emotion  and  control.  Deep  involvement  is  a psychological  state 


256 


lADIS  International  Conference  e-Learning  2013 


that  occurs  when  the  related  set  of  activities  or  events  in  the  game  are  solely  in  focus  (Witmer  and  Singer, 
1998).  Players  become  oblivious  to  their  surrounding  (Johnson  and  Wiles,  2003;  Brown  and  Cairns,  2004), 
and  lose  their  sense  of  time  and  concern  over  everyday  life  (King  and  Schell,  2006).  Yet,  involvement  is 
crucial  for  learning.  It  can  be  used  to  transfer  knowledge  and  information  efficiently  (Csikszentmihalyi, 
1990).  The  other  factor  that  leads  to  immersion  is  emotion.  Players  could  sometimes  be  emotionally  affected 
by  their  progress  in  the  game  (Brown  and  Cairns,  2004).  Narration,  sound  (Sweetser  and  Johnson  2004)  and 
features  of  fantasy  (Li  et  ak,  2013)  are  elements  that  endear  players  to  the  game.  Furthermore,  immersion  is 
not  achievable  unless  players  have  appropriate  control  over  the  game.  The  game  control  should  be  simple  to 
learn  and  easily  customizable  (Gee,  2003;  Johnson  and  Wiles  2003). 

Challenge  refers  to  the  differential  between  a player’s  skills  and  the  current  demands  imposed  by  the 
game.  Existence  of  a variety  of  difficulty  levels,  clear  goals  and  appropriate  rewards  are  necessary 
ingredients  for  a game  to  be  deemed  challenging.  Games’  difficulty  levels  should  be  varied  and  gradually 
increased  based  on  players’  development  to  keep  their  interest.  Appropriate  pacing  is  required  to  keep  the 
players  engaged  (Pagulayan  et  ak,  2003).  Furthermore,  completing  difficult  tasks  and  surpassing  opponents 
satisfy  the  players  and  lead  to  pleasure  and  a sense  of  accomplishment  (Vorderer  et  ak,  2003;  Poyrazli  and 
Grahame,  2007;  Lazzaro,  2009).  Also,  each  level  of  game  should  have  its  own  clear  goals  (Csikszentmihalyi, 
1990;  Pagulayan  et  ak,  2003).  Moreover,  players  must  be  rewarded  sufficiently  and  equitably  to  their 
expended  effort  to  sustain  their  interest  in  the  game  (Pagulayan  et  ak,  2003;  Brown  and  Cairns,  2004). 

Social  interaction  is  another  factor  that  contributes  to  fun.  In  the  context  of  a game,  social  interaction 
takes  the  form  of  competition,  cooperation  and  connection  (Lazzaro,  2009),  each  of  which  caters  to  a 
different  player  profile.  Players  with  a strong  desire  to  win  or  hinder  other  players  from  winning  will  find 
competitive  elements  in  the  game  gratifying  (Brathwaite  and  Schreiber,  2008).  Players  who  prefer  to  work 
together  to  improve  their  standing  in  the  game  will  find  cooperative  elements  appealing  (Salen  and 
Zimmerman,  2004).  Players  who  engage  in  a game  not  solely  for  gameplay  but  for  social  reasons  likely  to 
enjoy  the  connection  forged  with  fellow  players  both  inside  and  outside  of  the  game  (Lazzaro,  2009). 

Table  1.  Fun  framework 


Fun  factors 

Dimensions 

Description 

Immersion 

Deep 

involvement 

■ Psychological  state  happened  by  focusing  (Witmer  and  Singer,  1998) 

■ Oblivious  to  their  surrounding  environment  (Johnson  and  Wiles,  2003;  Brown  and 
Cairns,  2004) 

■ Loss  concern  without  effort  (Klug  and  Schell,  2006) 

Emotion 

■ Affected  by  the  player  progress  in  the  game  (Brown  and  Cairns,  2004) 

■ Affected  by  narration,  sound  (Sweetser  and  Johnson,  2004)  and  features  of  fantasy 
(Li  etak,  2013) 

Control 

■ Basic  controls  features  (Gee,  2003;  Johnson  and  Wiles,  2003) 

■ Easily  customizable  (Gee,  2003;  Johnson  and  Wiles,  2003) 

Challenges 

Difficulty 

level 

■ Different  challenges’  level  based  on  player  progress  (Pagulayan  et  ak,  2003) 

■ New  challenges  at  an  appropriate  Pace  (Pagulayan  et  ak,  2003) 

■ Satisfaction  from  accomplishing  difficult  level  and  surpassing  opponents  (Vorderer 
et  ak,  2003;  Poyrazli  and  Grahame,  2007;  Lazzaro,  2009) 

Goals 

■ Clear  goals  (Csikszentmihalyi,  1990;  Pagulayan  et  ak,  2003) 

■ Multiple  goals  in  each  level  (Csikszentmihalyi,  1990;  Pagulayan  et  ak,  2003) 

Rewards 

■ Efforts  should  be  rewarded  (Brown  & Cairns  2004;  Pagulayan  et  ak,  2003) 

■ Encourage  players  to  continue  (Brown  and  Cairns,  2004;  Pagulayan  et  ak,  2003) 

Social 

interaction 

Competition 

■ Some  players  enjoy  fighting  and  beating  each  other  (Lazzaro,  2009) 

■ It  is  an  effective  motivation  to  continue  playing  (Brathwaite  and  Schreiber,  2008) 

Cooperation 

■ Emphasize  is  on  fun  rather  than  competition  (Salen  and  Zimmerman,  2004) 

■ Work  toghether  to  improve  their  standing  in  the  game  (Salen  and  Zimmerman,  2004) 

Connection 

■ Some  players  play  not  solely  for  gameplay  but  for  social  reasons  (Lazzaro,  2004) 

■ Social  connection  inside  and  outside  the  game  (Lazzaro,  2009) 
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3.  DGIST  - DESIGN  OVERVIEW 

DGIST  (Digital  Game  for  International  Student  Training)  is  a desktop  online  multiplayer  game  built  on 
design  principles  related  to  international  students’  information  needs  and  the  fun  aspects  of  games.  It  is 
designed  to  combine  learning  and  entertainment.  The  scenario  of  DGIST  is  the  story  of  an  international 
student  (represented  by  an  avatar)  who  comes  to  a host  country  (Singapore)  to  study  at  a local  university. 
During  this  journey,  the  avatar  is  required  to  navigate  in  a virtual  environment  to  acquire  adjustment’s 
information. 

The  main  goal  of  DGIST  is  to  facilitate  international  students  acquire  adjustment-related  information 
through  fun.  The  game  features  six  different  spaces,  namely.  Location,  Food,  History,  Culture,  Education  and 
Social  Networking.  These  spaces  are  intended  to  meet  the  four  adjustment-related  information  needs,  as 
shown  in  Table  2. 


Table  2.  Spaces  in  DGIST 


Spaces  in  DGIST 

Description 

Adjustment-related  information  needs 

Location 

■ Introduces  important  locations 

■ Shows  the  way  they  can  be  reached 

Preliminary  Needs 

Food 

■ Introduces  local  food  and  drinks 

■ Teaches  how  to  prepare  local  foods  and  drinks 

History 

■ Offers  Singapore  historical  information  by  narration 

■ Asks  historical  questions 

Cultural  Needs 

Culture 

■ Introduces  some  aspects  of  Chinese,  Indian  and 
Malay  cultures 

■ Introduces  some  local  idiom  and  phrases 

Education 

■ Introduces  the  local  university  facilities  and  rules 

Educational  needs 

Social  Networking 

■ Allows  virtual  clubs  to  be  created 

■ Supports  connection  via  chatting  tool  or  bilboard 

■ Allows  competition  and  cooperation  in  virtual  clubs 

Psychological  needs 

Difficulty  level  of  each  space  is  increased  gradually  based  on  the  progress  of  the  player.  Figure  1 shows 
some  screen  shots  of  DGIST.  A brief  description  of  each  space  is  as  follows. 


(a)  (b)  (c) 


Figure.  1.  (a)  DGIST’ s main  menu  (b)  Jigsaw  Puzzle  in  Location  space  (c)  Question/answer  in  History  space 

The  DGIST’ s Location  space  is  designed  to  familiarize  players  with  important  locations  in  Singapore  and 
how  they  can  reach  there.  In  this  space  players  are  presented  with  a set  of  Jigsaw  puzzles  and 
question/answer  games.  For  example,  to  learn  how  to  get  to  the  Immigration  & Checkpoint  Authority  (ICA), 
players  solve  the  Mass  Rapid  Transit  (MRT)  jigsaw  puzzle  to  know  where  the  nearest  MRT  station  to  ICA  is. 

To  make  international  students  familiar  with  the  host  country's  food,  DGIST  introduces  some  local  food 
and  drinks.  In  Food  space,  players  learn  how  to  prepare  local  food  and  drinks  by  selecting  their  ingredients 
and  adding  them  to  pot  or  solving  word  puzzle  to  be  acquainted  with  the  names  of  local  food. 

History  space  offers  Singapore's  historical  information  through  narration  and  question/answer  game.  For 
instance,  though  having  a student’s  pass  is  essential  for  international  students  to  stay  in  Singapore;  the  player 
is  guided  how  to  get  it  via  narration.  He/she  should  find  out  where  to  go  and  answers  historical  question  to 
gain  enough  points  and  get  the  student  pass  as  a reward. 
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Singapore  is  a multi-racial  society  which  comprises  three  distinct  ethnic  groups:  Chinese,  Indian  and 
Malay.  In  the  DGIST’s  Culture  space,  players  become  familiarize  with  some  salient  aspects  of  these  three 
cultures,  for  instance  the  appropriate  form  of  greeting  or  gift-giving.  Although  English  is  the  official 
language  in  Singapore,  some  idiom  and  phrases  used  locally  are  derived  from  Chinese  or  Malay  languages. 
The  most  common  use  of  these  phrases  and  their  meaning  are  also  covered  in  Culture  space.  In  addition, 
since  DGIST  was  designed  for  a local  university,  specific  university’s  facilities  and  rules  are  introduced  in 
Educational  space. 

Social  interaction  during  the  gameplay  is  facilitated  through  chatting  and  virtual  communities.  New 
players  are  required  to  enter  some  information  such  as  program  of  study  and  nationality.  Thereafter,  players 
can  create  a virtual  club  and  named  it  based  on  their  countries  or  program  of  study  to  make  a domicile  for 
others  with  similar  profiles.  DGIST's  virtual  clubs  serve  as  a social  platform  that  allows  students  to  plan 
activities  in  its  virtual  environment  or  in  the  real  world.  For  instance,  club  members  can  set  an  online  chess 
competition  or  arrange  playing  basketball  at  weekend  in  the  school's  court.  All  club  members  can  create 
preferred  event  on  the  billboard  while  others  can  add  further  comments  on  these  events.  Players  can  register 
in  different  clubs.  To  encourage  the  expansion  of  social  networks,  DGIST  allocates  points  for  inviting  friends 
to  the  clubs.  Each  club  shows  its  online  and  offline  members. 

Mentioned  earlier,  the  fun  framework  is  considered  in  the  design  of  DGIST  to  make  it  entertaining.  For 
instance,  jigsaw  puzzle  game,  question/answer,  cooperation  and  competition  in  virtual  clubs  have  been 
incorporated  into  DGIST  to  create  an  immersive  experience  for  players.  Narration  and  sound  effects  are  used 
as  features  to  influence  emotion.  Also,  DGIST  offers  the  sense  of  control  by  letting  players  choose  their 
avatars,  the  activities  they  prefer  to  be  involved  in  their  virtual  club  and  the  part  of  the  game  they  want  to 
proceed. 

Challenges,  another  factor,  is  achieved  in  DGIST  by  defining  different  levels  of  difficulty  with  various 
goals  in  each  space.  For  example,  in  the  basic  level  question/answer  game  offers  multiple  choices  while  in 
more  advanced  level,  players  are  required  to  submit  their  answers  in  free-text  format.  Also,  rewards  players 
receive  in  DGIST  commesurate  with  the  different  levels  of  difficulty. 

Based  on  the  fun  framework  social  interaction  is  another  key  element  that  leads  to  enjoyment  in  the  game. 
Social  Network  space  of  DGIST  is  designed  for  this  reason,  and  its  chatting  tool  and  virtual  clubs  with 
bilboards  are  used  to  create  competition,  cooperation  and  social  connection  between  players.  Table  3 
illustrates  how  fun  factors  have  been  incorporated  into  DGIST. 


Table  3.  Fun  framework  & corresponding  DGIST  features 


Fun  factors 

Dimensions 

DGIST’s  Features 

Immersion 

■ Deep  involvement 

■ Puzzle  game,  question/answer,  cooperation/  competition  in  virtual  clubs 

■ Emotion 

■ Narration  and  sound  effects 

■ Control 

■ Choosing  avatars,  activities  in  virtual  club,  ingame  navigation 

Challenges 

■ Difficulty  level 

■ Different  levels  of  challenges  that  increased  gradually 

■ Goals 

■ Various  goals  in  each  level 

■ Rewards 

■ Specific  rewards  that  are  designed  for  each  level. 

Social 

■ Competition 

■ Chatting  tool 

interaction 

■ Cooperation 

■ Virtual  clubs 

■ Connection 

■ Bilboards 

4.  METHODOLOGY 

A total  of  80  graduate  international  students  in  a local  university  in  Singapore  were  invited  as  part  of  the 
study.  With  an  average  age  of  26  years  old,  all  had  arrived  in  Singapore  for  less  than  a year.  They  came  from 
nine  countries,  including  China,  India,  Iran,  Spain,  Italy,  Sweden,  Indonesia,  Kazakhstan  and  Myanmar. 

Before -and- After  with  control  experimental  design  was  used  to  evaluate  the  effectiveness  of  DGIST 
(Kothari,  2004).  The  80  participants  were  randomly  and  equally  divided  into  the  control  and  experimental 
group.  In  the  pre-test,  all  80  participants  in  both  groups  were  asked  to  answer  a questionnaire  which 
comprises  39  multiple-choice  questions  about  Singapore’s  location,  food,  history,  culture  as  well  as 
University’s  rules  and  facilities.  All  questions  have  only  one  correct  answer.  The  purpose  of  the  pre-test  was 
to  assess  participants’  familiarity  with  adjustment-related  information  before  treatment. 
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Thereafter,  during  the  treatment  phase,  participants  in  the  control  group  were  given  a six  pages  paper- 
based  document  which  contained  a combination  of  text  and  images,  not  unlike  a typical  travel  guidebook.  It 
includes  the  required  information  to  answer  all  the  questions  in  the  questionnaire.  In  contrast,  participants  in 
the  experimental  group  played  DGIST  to  acquire  the  same  information.  Based  on  a pilot  study  which  was 
done  before  post-test,  five  minutes  was  deemed  sufficient  to  scan  the  document  or  play  DGIST.  Thus,  the 
treatment  phase  lasted  for  five  minutes  for  all  participants  to  ensure  that  both  groups  were  exposed  to  similar 
conditions  in  the  study  (Kothari,  2004). 

In  the  post-test,  participants  in  both  groups  were  requested  to  revisit  the  questionnaire,  and  amend  any 
answers  they  felt  to  be  erroneous.  However,  control  group  participants  were  not  allowed  to  refer  back  to  the 
paper-based  document  while  experimental  group  participants  were  denied  access  to  DGIST.  To  differentiate 
markings  made  during  the  pre-test  and  the  post-test,  participants  used  a different  colored  pen.  To  capture 
participants’  experiences  from  the  control  and  experimental  group,  qualitative  feedback  was  collected  via 
face-to-face  interviews.  Specifically,  participants  were  asked  how  they  were  able  to  retain  adjustment-related 
information  and  what  improvement  could  be  made  to  enhance  the  acquisition  of  such  information. 

To  compare  the  effectiveness  between  the  paper-based  document  and  DGIST  in  helping  participants 
acquire  adjustment-related  information,  the  paired  t-test  was  used  to  check  for  any  differences  between  the 
improvements  of  the  two  groups. 


5.  RESULTS  AND  DISCUSSION 


5.1  Analysis  of  Pre-Test  and  Post-Test  Scores 

Scores  obtained  from  all  participants  were  normalized  to  100  for  easy  computation.  In  the  pre-test,  the  mean 
score  of  the  control  group  was  44.29  + 13.24  and  that  for  the  experimental  group  was  44.35  + 13.99  At  5% 
level  of  significance,  there  was  no  statistical  difference  between  the  two  group.  This  means  that  participants 
in  both  groups  were  comparable  in  terms  of  their  knowledge  on  adjustment-related  information. 

In  the  post-test,  the  control  group  obtained  a mean  score  of  57.88  + 15.78  points.  The  mean  improvement 
is  13.59  + 8.33.  At  5%  level  of  significance,  the  pre-test  scores  and  the  post-test  scores  of  the  control  group 
were  found  to  have  unequal  variances  (t(39)=10.31,  p < 0.001).  This  indicates  that  the  paper-based  document 
had  been  effective  in  helping  participants  in  the  control  group  acquire  adjustment-related  information. 

The  experimental  group,  however,  obtained  a markedly  higher  mean  score  of  84.16  + 12.03  in  the  post- 
test. The  mean  improvement  is  39.81+12.48.  At  5%  level  of  significance,  the  pre-test  scores  and  the  post-test 
scores  of  the  experimental  group  were  found  to  have  unequal  variances  (t(39)=20.15,  p < 0.001).  This 
indicates  that  DGIST  had  been  effective  in  the  experimental  group  to  acquire  adjustment-related  information. 

To  compare  the  efficiency  of  document  and  game  based  training,  the  improvements  in  scores  of  all 
participants  in  each  group  were  evaluated.  The  mean  of  this  improvement  in  control  group  is  13.59  and  in 
experimental  group  it  is  39.81.  At  5%  level  of  significance,  the  improvement  scores  between  the  control 
group  and  that  of  the  experimental  group  were  found  to  have  unequal  variances  (t(78)=l  1.95,  p < 0.001). 
This  shows  that  the  effectiveness  of  the  two  approaches  is  significantly  different.  In  particular,  DGIST 
appeared  to  be  more  effective  compared  to  the  paper-based  document  approach  in  helping  international 
students  acquire  adjustment-related  information. 

5.2  Qualitative  Feedback  from  Participants 

Fun  could  be  one  of  the  features  of  DGIST  that  contributed  to  its  effectiveness.  The  participants  in 
experimental  group  were  able  to  acquire  information  easily  through  gameplay.  For  example.  Participant  1 
indicated  that  “if  was  helpful  and  fun,  I won’t  forget  the  information  I learned  and  it  was  much  better  than 
reading  stuff’.  Moreover,  social  interaction  in  DGIST  is  the  way  of  overcoming  psychological  challenges. 
Participant  10  mentioned  that  “being  part  of  the  DGIST’ s virtual  communities  helps  to  create  sense  of 
belongingness” . The  graphic,  sound  effects  and  fun  aspects  of  DGIST  enabled  players  to  be  deeply  involved 
in  the  game.  For  example.  Participant  28  smiled  after  he  saw  the  Malay  lady  character  in  Culture  space  and 
commented  that  “this  cute  Malay  lady  helps  me  to  distinguish  Malay  women  easier”. 
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When  asked  what  improvements  could  be  made  to  DGIST,  a few  suggestions  were  made.  For  instance, 
some  players  mentioned  that  after  they  acquired  information  on  DGIST,  they  were  less  motivated  to  persist 
in  the  game.  In  particular.  Participant  15  mentioned  that  "I  wish  to  see  different  questions  each  time  I play 
question/answer  game".  This  was  due  to  the  limited  set  of  questions/answer  at  each  level  of  difficulty  in 
DGIST.  Next,  players  suggested  that  DGIST  be  deployed  as  a mobile  game.  Participant  35  indicated  "I  want 
to  have  DGIST  on  my  mobile  devices  to  play  it  while  I am  on  the  way  to  school".  Another  suggestion  was  to 
allow  for  greater  interaction  among  players  during  gameplay.  Participant  17  mentioned  that  "I  wish  I could 
input  comments  in  each  space  and  share  my  experiences  about  how  to  answer  questions  and  do  puzzles  with 
my  friends". 

While  the  document-based  method  was  also  effective  in  helping  participants  acquire  adjustment-related 
information,  it  may  be  difficult  to  sustain  attention  from  participants.  Most  of  the  participants  in  control 
group  showed  little  interest  to  read  the  document  and  found  it  boring.  For  instance.  Participant  48  said  “this 
document  reminds  me  of  my  exam  days".  When  asked  on  the  improvements  to  be  made  to  enhance 
information  acquisition,  participants  were  generally  averse  to  the  use  of  paper-based  document.  In  fact,  as 
digital  devices  become  ubiquitous,  using  a paper-based  document  was  considered  a nightmare.  Participant 
61  commented  that  “I  really  miss  Ctr+F  function  key". 

Thus,  although  both  paper-based  document  and  DGIST  could  be  used  for  adjustment-related  information 
acquisition,  the  latter  was  found  to  be  more  effective.  In  particular,  the  entertainment  features  in  DGIST  not 
only  sustained  participants’  interests,  but  contributed  to  better  information  retention. 


6.  CONCLUSION 

To  help  international  students  cope  with  adjustment  issues,  this  paper  seeks  a two-fold  objectives.  The  first  is 
to  introduce  DGIST  as  a means  to  satisfy  important  aspects  of  international  students’  information  needs 
through  fun.  The  second  is  to  perform  a preliminary  evaluation  of  DGIST  in  terms  of  its  efficacy  in  helping 
international  students  acquire  adjustment-related  information.  To  address  the  first  objective,  DGIST  was 
developed  on  the  basis  of  the  literature.  The  game  features  six  different  spaces,  namely.  Location,  Food, 
History,  Culture,  Education  and  Social  Networking,  which  are  intended  to  meet  the  preliminary  needs, 
cultural  needs,  educational  needs  and  psychological  needs  of  international  students.  To  address  the  second 
objective,  a before-and-after  with  control  experimental  design  was  used  to  compare  the  efficacy  between 
DGIST  and  a paper-based  document.  A total  of  80  participants  were  involved  in  the  study.  Despite  a few 
shortcomings,  the  results  for  DGIST  appeared  promising.  Statistical  analyses  confirmed  that  DGIST  was 
more  effective  in  helping  students  acquire  adjustment-related  information  than  the  paper-based  document. 

Nonetheless,  some  limitations  should  be  acknowledged  in  this  study.  First,  the  findings  must  be 
interpreted  in  light  of  the  limited  sample  size  of  80  participants.  The  second  limitation  is  related  to  DGIST. 
Its  design  was  confined  to  Singapore's  culture  and  one  of  its  local  universities.  Thus,  it  is  only  applicable  to  a 
specific  context. 

Therefore,  expanding  the  number  of  participants  and  inviting  more  international  students  from  different 
universities  can  be  considered  as  a future  work  to  make  the  results  more  reliable.  The  other  possible 
extension  to  the  current  work  is  to  broaden  DGIST's  design  to  cover  more  cultures  and  educational  rules.  In 
addition,  knowledge  sharing  functionalities  can  be  incorporated  into  DGIST  to  encourage  players  to  be  more 
helpful  and  interactive.  Also,  increasing  the  complexity  of  the  game  and  adapt  it  based  on  players  proficiency 
level  will  be  considered  as  a future  work  to  make  it  more  attractive  for  players. 
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ABSTRACT 

This  paper  reports  the  findings  of  a study  which  seeks  to  identify  the  information  and  communications  technology  (ICT) 
literacy  levels  of  trainee  teachers,  by  investigating  their  ICT  proficiency  using  a task-bask  assessment  instrument.  The 
Delphi  technique  was  used  as  a primary  validation  method  for  the  new  assessment  tool  and  the  ICT  literacy  task-based 
assessment  tool  was  developed  to  assess  trainee  teacher’s  levels  of  ICT-literacy.  The  findings  showed  that  the 
ICT-literacy  task-based  assessment  instrument  were  able  to  predict  trainee  teacher’s  ICT-literacy  area  of  weaknesses. 
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1.  INTRODUCTION 

The  central  concern  of  this  study  is  to  develop  an  ICT-literacy  task-based  assessment  (TBA)  instrument  that 
may  be  used  to  evaluate  trainee  teacher’s  level  of  ICT-literacy.  The  current  literature  acknowledges  the  need 
for  a measurement  instrument  that  evaluates  level  of  ICT-literacy  levels  (Wong  2002;  Markauskaite  2005; 
Katz  & Macklin  2007). 

This  type  of  measurement  instrument  is  currently  used  as  an  entry  level  testing  tool  for  university  and  job 
placements.  However,  existing  ICT-literacy  assessment  instruments  are  either  too  expensive  to  be 
implemented,  or  too  rigid  with  their  expected  answers.  Moreover,  they  are  not  tailored  to  teacher’s  individual 
needs.  The  existing  instruments  either  uses  self-efficacy  techniques  or  step-by-step  task/instructions  whereby 
they  do  not  allow  flexibility  and  creativity  in  completing  the  task  (International  ICT  Literacy  Panel  2002; 
Cox  & Marshall  2007). 

Following  the  inauguration  of  the  International  ICT-literacy  Panel  in  2000,  the  International  Society  for 
Technology  in  Education  (ISTE)  in  2008,  proposed  that  in  order  to  accurately  assess  level  of  ICT-literacy, 
such  an  assessment  must  include  both  technical  and  information  literacy  (International  ICT  Literacy  Panel 
2002;  International  Society  for  Technology  in  Education  2008).  Two  ICT  standards  which  are  the 
Information  Literacy  Competency  Standards  for  Higher  Education  developed  by  the  Association  of  College 
and  Research  Libraries  (ACRL)  and  the  Australian  and  New  Zealand  Information  Literacy  (ANZIIL)  also 
echo  the  views  of  the  2000  International  ICT-literacy  Panel  and  2008  ISTE  (Association  of  College  and 
Research  Libraries  2000;  ANZIIL  2008). 


2.  WHY  TRAINEE  TEACHERS? 

Currently,  ICT-literacy  has  been  actively  promoted  in  Malaysian  schools  by  various  agencies  of  the 
Malaysian  Ministry  of  Education.  The  Ministry  has  also  made  it  compulsory  for  all  trainee  teachers  to  be 
exposed  to  ICT  tools,  and  by  implication,  the  use  of  ICT-literacy  in  their  pedagogical  strategies  (Chan 
2002b). 
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The  project  team  who  was  responsible  for  the  ICT-implementation  in  Malaysian  schools  has  listed  three 
abilities  expected  of  Malaysian  students.  This  abilities  include  competencies  to  use  ICT  tools  and  ICT 
sources  to:  1)  collect,  analyse,  process,  and  present  information;  2)  support  meaningful  learning  in  various 
contexts;  and  3)  prepare  students  for  employment  (Smart  School  Project  Team  1997).  The  competencies 
listed  by  the  team  coincide  with  the  definition  of  technology  literacy  described  by  NAE  and  NRC;  which 
called  for  an  understanding  of  technology  at  a level  that  enable  effective  functioning  in  a modern 
technological  society  (NAE  & NRC  2006).  In  focusing  on  improving  school  children’s  ICT-literacy,  it  is 
possible  that  current  Malaysian  trainee  teachers  may  not  be  adequately  prepared  to  teach  under  this  new 
approach. 

In  spite  of  this,  ICT  tools  must  never  be  mistaken  as  the  mechanism  that  we  learn  ‘from’,  but  rather  as  the 
tool  that  we  leam  ‘with’.  By  having  ICT  tools  such  as  computers,  for  example,  they  should  be  regarded  as  a 
learning  aid  rather  than  a learning  point.  Yet,  in  looking  at  the  current  Malaysian  school  scenario,  Chan 
(2002a)  identifies  a serious  gap  between  what  is  being  understood  by  the  teachers  and  what  they  actually 
practice.  Most  teachers  prefer  to  use  ICT  to  enhance  rather  than  transform  their  current  curriculum.  Rather 
than  changing  their  old  teaching  module  to  include  ICT  in  their  teaching  activities,  most  teachers  felt  that  it 
would  be  easier  to  use  ICT  to  accommodate  and  enhance  their  existing/older  teaching  modules.  Eor  example: 
instead  of  using  the  overhead  projector  (OHP),  the  teachers  would  input  their  teaching  notes  into  a digital 
presentation  application  (for  example  MS  Powerpoint);  or  instead  of  manually  creating  a class  timetable,  the 
teachers  are  now  using  spreadsheets.  It  is  proposed  that  these  teachers  may  not  have  the  initiative  or  time  to 
explore  other  aspects  of  ICT  that  they  could  use  to  improve  their  teaching  activities  (BECTA  June  2004). 

In  2004,  a group  from  the  National  Academy  of  Engineering  (NAE)  and  the  National  Research  Council 
(NRC)  conducted  a study  to  determine  the  most  viable  approach  to  assessing  technological  literacy  in  the  US 
for  K-I2  students,  K-I2  teachers,  and  out-of-schooI  adults.  The  report  found  that  there  was  very  little 
information  available  on  the  technological  literacy  of  teachers  (NAE  & NRC  2006).  Although  many  school 
children  have  sophisticated  technological  capabilities,  they  cannot  be  fully  technologically  literate  unless 
their  teachers  are.  There  is  an  urgent  need  for  an  in-depth  study  on  this  topic  and  development  of  a suitable 
task-based  (technological  capability)  assessment  instrument. 


3.  THE  NEED  EOR  A DIEEERENT  INSTRUMENT  TO  ASSESS  TRAINEE 
TEACHERS’  ICT-LITERACY 

Several  studies  have  been  conducted  on  students  (Katz  2007;  Russell  & Einger  2007),  trainee  teachers,  and 
in-service  teachers  (Luke  2001;  Knezek  & Christensen  2002;  Jamieson-Proctor,  Burnett,  Linger  & Watson 
2006).  Most  of  them  involve  participants’  perceptions  and  attitudes  on  their  preparedness  to  integrate  ICT  as 
tools  or  teaching  ICT  as  the  class  subject. 

The  quality  of  the  research  design  in  ICT-literacy  studies  is  also  an  issue  because  too  many  research 
studies  applied  the  self-assessment  methodologies  (see  Jamieson-Proctor,  Burnett,  Linger  & Watson  2006; 
Markauskaite  2007;  Ball  & Levy  2008).  Computer  self-efficacy  involves  a belief  of  one’s  capability  to  use  a 
computer.  Markauskaite’ s (2007)  study  for  example,  utilised  the  self-efficacy  theory  in  her  questionnaire 
design.  Each  test-item  in  her  questionnaire  started  with  the  phrase  “I  believe  I have  the  capability. . .”  and  was 
measured  using  a six-point  Likert  scale. 

Other  studies  applied  a combination  of  both  self-efficacy  methodology  and  a more  hands-on  evaluation  of 
the  participants  ICT-literacy.  One  such  example  was  in  Wong  (2002)  study.  In  her  study,  she  developed  an 
IT  preparedness  assessment  instrument,  which  measured  teachers’  preparedness  in  using  ICT  using  three 
different  measures:  a self-efficacy  instrument  to  evaluate  their  attitude  towards  using  ICT;  a hands-on 
instrument  to  assess  their  ICT  skills;  and  ICT-knowIedge  exam-based  questions  that  consists  of  25  multiple 
choice  questions  which  tested  the  teachers  knowledge  on  ICT  and  computers.  Nonetheless,  the  hands-on 
instrument  lacks  flexibility  and  does  not  encourage  critical  and  analytical  thinking,  as  it  gives  the  teachers 
step-by-step  instructions  of  what  was  to  be  performed,  and  the  exam-based  questions  may  not  represent  the 
teachers’  actual  ICT  knowledge  as  the  multiple  choice  questions  may  allow  guessing. 

To  cope  with  today’s  technological  demands,  people  need  to  acquire  more  than  just  the  basic  ICT  skills 
and  knowledge.  They  need  to  know  how  to  use  their  acquired  knowledge  and  skills  by:  thinking  critically; 
applying  knowledge  to  new  situations;  analysing  information;  generating  new  ideas;  communicating; 
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collaborating;  solving  problems;  and  making  decisions.  These  skills  can  provide  both  flexibility  and  security. 
People  who  can  learn  new  information,  are  able  to  use  software  programs  and  conceive  new  ways  of  doing 
things,  have  much  better  prospects  than  those  who  cannot  (Partnership  for  the  21st  Century  Skills  2002). 

The  lack  of  ability  to  think  critically  and  analytically,  and  also  to  make  decisions  is  apparent  in  Malaysian 
school  students.  Earlier  studies  have  shown  that  students’  critical  and  analytical  thinking  abilities  in  Malaysia 
were  between  below  satisfactory  and  fair  (Zaharah  1995;  Azlan  1997;  Razali  1999). 

It  is  further  noted  that  Malaysian  teachers  may  also  lack  the  ability  to  teach  these  skills  or  are  less 
prepared  to  teach  by  incorporating  these  skills  in  teaching  and  learning  activities  (Rajendran  2001;  Rosnani 
2002).  Teachers  may  understand  the  importance  of  teaching  critical  thinking  to  students,  yet  some  appear  to 
not  have  the  necessary  instructional  strategies  to  teach  it  (Rosnani  2002).  These  issues  corroborate  this 
study’s  need  for  an  enhanced  ICT-literacy  TEA  instrument  that  is  flexible;  not  based  on  self-assessment;  and 
includes  test-items  that  test  cognitive  skills.  The  tasks  for  the  ICT-literacy  TEA  instrument  should  focus  on 
familiar/normal,  computer-based  activities  for  a classroom  environment  that  teachers  usually  find  in  their 
schools. 

A task-based  assessment  method,  allows  the  participants  the  freedom  to  complete  the  task  in  any  way 
they  wish,  as  long  as  the  task  requirement  is  fulfilled.  It  means  that,  if  the  task  asks  for  an  appropriate 
learning  aid  to  be  created  which  includes  an  image  and  a video,  the  participant  is  free  to  use  whatever 
computer  applications  that  they  feel  comfortable  to  use,  to  edit  the  pictures,  create  videos  and  other  digital 
learning  aids.  As  long  as  the  task  requirement  is  fulfilled,  the  task  is  considered  complete.  Task-based  tests 
allow  the  participant  to  demonstrate  acquired  knowledge  capacity  and  an  ability  to  correlate  tasks  with  the 
theories  or  concepts  learned  previously.  Instead  of  judging  knowledge  acquisition  through  a series  of 
multiple  choice  selections  or  self-evaluation,  task-based  assessment  forces  participants  to  place  knowledge 
into  a context  that  can  be  understood  and  explained  (Teachnology  Inc  2011).  Task-based  assessment  also 
allows  the  participants  to  show  what  they  know,  instead  of  just  telling  what  they  think  they  know.  It  is 
considered  as  the  best  method  for  this  new  ICT-literacy  assessment  instrument  as  it  shows  the  participant’s 
actual  ICT  ability. 


4.  THE  METHODOLOGY 

This  study  was  conducted  in  three  phases:  Phase- 1 preliminary  review;  Phase-2  expert  judgement  on  ICT- 
literacy  indicators;  and  Phase-3  instrument  validation  and  testing.  In  Phase- 1,  review  of  the  literature  was 
conducted,  where  it  involves  drawing  on  the  existing  literature  on  ICT-literacy;  ICT-literacy  standards; 
existing  ICT-literacy  assessment  instruments;  and  the  Malaysian  Smart  School  (MSS)  requirements.  Eleven 
ICT-literacy  indicators  were  identified  in  this  first  research  phase.  In  Phase-2,  the  identified  ICT-literacy 
indicators  were  evaluated  by  a specially  chosen  panel  of  experts  (PoE).  Two  Delphi  interactions  were  then 
conducted  where  the  first  was  to  evaluate  the  ICT  indicators,  and  the  second  was  to  validate  the  draft 
ICT-literacy  TBA  instrument.  In  Phase-3,  the  draft  ICT-literacy  TBA  instrument  was  validated  and  tested 
through  two  pilot  tests,  and  finally  the  instrument  was  tested  on  a larger  number  of  participants  for  its  final 
instrument  trial. 

The  Delphi  technique  and  the  Rasch  IRT  model  were  applied  for  the  qualitative  and  quantitative  parts  of 
this  study,  respectively.  The  Delphi  technique  was  chosen  as  the  most  suitable  philosophical/research 
approach  for  conducting  the  expert  judgement  phase  (Phase-2)  of  this  study. 

Eor  the  quantitative  part  of  this  study,  the  Rasch  IRT  model  was  employed.  The  Rasch  IRT  model  is 
assumed  to  be  invariant  across  different  groups  within  a research  population  and  across  populations 
(Hambleton  & Murphy  1991;  Swaminathan  1999).  It  requires  that  both  the  test-items  and  participants 
conform  to  the  Rasch  IRT  model  before  claims  regarding  the  presence  of  skill  or  ability  can  be  considered 
valid.  Therefore,  under  this  Rasch  IRT  model,  mis-fitting  responses  require  a reason  for  the  misfit,  and  may 
be  excluded  from  the  data  set  if  they  fail  to  address  the  expected  skill  or  ability.  The  Rasch  IRT  model  also 
provides  a way  of  measuring  the  quality  of  the  test-items  by  confirming  their  suitability  for  participants  and 
how  well  they  measured  participants’  abilities  (Wu  & Adams  2007). 

The  Rasch  IRT  model  generally  utilises  the  response  pattern.  It  assumes  that  participants  with  a low 
attribute  have  little  chance  of  guessing  the  correct  answer  and  participants  who  achieve  a high  attribute  will 
almost  certainly  choose  the  correct  answer  (Nunnally  & Eernstein  1994).  Central  to  idea  of  the  Rasch  IRT 
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model  is  the  probability  principle.  A person’s  response  to  a particular  test-item  is  never  certain.  It  is  always 
influenced  by  human  error.  Thus,  a probabilistic  approach  must  be  employed.  In  the  Rasch  IRT  model, 
probabilities  are  introduced  through  consideration  of  the  odds  that  a person  would  give  a correct  response  to 
a test-item.  This  is  known  as  logit.  The  logit  is  a mathematical  model  that  converts  both  difficulty  and  ability 
into  the  same  units.  The  logit  is  a loge  of  the  odds  of  a correct  response  given.  The  Rasch  dichotomous  model 
equation  used  in  the  Quest  analysis  describes  the  probability  of  observing  a specific  score  Xni  as  (Adams  & 
Khoo  1996): 


3wxp{  - ^tl)) 
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Where  Xni  is  person  «’s  response  to  item  i,  is  the  ability  of  person  n,  Wn  is  the  score  assigned  to  one 
step  in  item  i,  and  Su  is  the  difficulty  of  the  one  step  in  item  /. 

The  validation  and  testing  process  showed  that  the  ICT-literacy  TBA  instrument  is  valid  and  reliable  when 
tested  on  its  intended  participants,  and  the  instrument  is  ready.  The  instrument  provides  information  with 
regard  to  each  participant’s  area  of  weakness  in  ICT. 


5.  THE  PARTICIPANTS 

There  were  two  categories  of  participants  for  this  study:  1)  the  qualitative  study  participants  (the  PoE 
members);  and  2)  the  quantitative  study  participants  (the  Malaysian  trainee  teachers). 

5.1  Qualitative  Study  Participants  - PoE  Members 

There  were  seven  PoE  participants  for  the  qualitative  study  and  they  were  selected  based  on  their  educational 
and  occupational  backgrounds.  The  PoE  members  were  three  academics  in  the  field  of  educational 
technology,  and  one  officer  from  the  Malaysian  Ministry  of  Education,  and  three  teachers  from  the  MSS. 

5.2  Quantitative  Study  Participants  - Malaysian  Trainee  Teachers 

One  of  Malaysia’s  public  universities,  the  Sultan  Idris  Education  University  (UPSI),  was  chosen  for  the 
location  to  conduct  the  research.  This  is  the  only  teachers’  university  in  Malaysia  with  the  sole  purpose  of 
training  pre-service  teachers.  Eor  this  study,  the  participants  were  undergraduate  students  who  were  currently 
enrolled  in  a Bachelor  of  Education  degree  at  UPSI.  The  university  has  eight  faculties  offering  both 
undergraduate  and  postgraduate  degrees.  The  students  are  generally  between  20  to  30  years  old,  and  of  mixed 
ethnicity  (Malay,  Chinese,  Indian,  Indigenous  and  others).  Participants  from  the  same  semester  across  eight 
different  faculties  were  invited  to  take  part. 

Different  participants  were  used  for  each  of  the  three  activities  (pilot  test-1,  pilot  test-2  and  final 
instrument  testing)  in  this  third  phase  of  the  ICT-literacy  TBA  instrument  development.  During  the  pilot 
testing- 1 session,  16  trainee  teachers  were  randomly  chosen  and  were  willing  to  participate.  Twenty  (out  of 
the  50  invited)  trainee  teachers  from  the  Eaculty  of  Business  and  Economics  participated  in  pilot  testing-2. 
Eor  the  final  instrument  testing  process,  148  (out  of  the  382  invited)  trainee  teachers  from  the  semester-four 
batch,  representing  all  faculties  in  the  university,  agreed  to  participate. 


6.  THE  FINDINGS 

There  were  three  parts  of  findings  which  was  based  on  the  different  phases  of  the  study.  As  previously 
explained.  Phase- 1 involved  identifying  suitable  ICT-literacy  indicators  to  be  used  to  evaluate  trainee 
teachers  level  of  ICT-literacy.  The  findings  from  Phase-2  and  Phase-3  are  discussed  below. 
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6.1  The  Findings  from  Phase-2 

The  seven  PoE  participants  reached  consensus  after  two  Delphi-rounds  have  been  conducted.  They  agreed 
that  there  were  twelve  ICT-literacy  indicators  that  should  be  used  to  evaluate  trainee  teachers’  level  of  ICT- 
literacy.  The  twelve  indicators  were: 


Table  1.  List  of  finalised  ICT-literacy  indicators 


Identified  ICT-literacy  indicators  : Finalised 

1. 

Navigation  and  search; 

7.  Manage; 

2. 

Production  and  analysis; 

8.  Create; 

3. 

Plan  / define; 

9.  Communicate  /collaborate; 

4. 

Access; 

10.  Reflect; 

5. 

Integrate; 

1 1 . Understanding  and  handling  ICT  tools. 

6. 

Evaluate; 

12.  Assess 

The  PoE  participants  also  validated  the  draft  ICT-literacy  TBA  instrument  where  the  ICT-literacy 
indicators  was  transformed  into  ICT-based  tasks  which  were  common  for  teachers.  For  evaluation  purpose, 
each  of  these  tasks  was  divided  into  a few  test-items. 


6.2  The  Findings  from  Phase-3 


The  draft  ICT-literacy  TBA  instrument  underwent  a continual  validation  and  testing  process  in  a series  of 
pilot  studies,  before  it  was  tested  on  a larger  population  of  trainee  teachers  in  the  final  instrument  testing 
process.  The  Rasch  IRT  model  and  the  Quest  interactive  test  item  analysis  system  (Adams  & Khoo  1996) 
was  implemented  as  the  data  analysis  tool  for  these  performance  data  validation  and  testing. 

For  the  final  instrument  testing,  all  148  participants  completed  all  six  tasks  in  the  ICT-literacy  TBA 
instrument.  The  responses  for  the  ICT-literacy  TBA  instrument  were  coded  by  the  researcher  into  an 
electronic  data  file  using  Microsoft  Excel.  A text  file  was  then  prepared  for  the  Quest  analysis  (Adams  & 
Khoo  1996). 

The  Quest  analysis  produces  a Kidmap  of  an  individual  participant’s  performance  in  terms  of  depicting 
their  correct  and  incorrect  response  patterns,  according  to  the  Rasch  IRT  model’s  expectations.  The  Kidmap 
locates  each  test-item  on  a vertical  scale  according  to  its  difficulty  from  easiest  to  hardest  and  then  separates 
the  test-items  horizontally  (left  or  right),  according  to  whether  the  participant  answered  them  correctly  or  not. 
Importantly  the  map  locates  the  participant’s  ability  on  the  same  vertical  scale. 


Figure  1.  Kidmap  - showing  an  individual’s  performance 
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Based  on  the  Kidmap  findings,  a graph  of  achievement  summary  of  all  the  candidates  was  produced  using 
the  data.  Figure  2 showed  the  level  of  the  participants  ICT  literacy  ability  based  on  their  achievement  for 
each  test-item.  For  each  test-item,  the  achievement  were  grouped  into  four  level  of  competency:  1)  Easier 
Achieved  (EA),  2)  Flarder  Achieved  (HA),  3)  Easier  Not  Achieved  (ENA),  and  4)  Harder  Not  Achieved 
(HNA). 


This  group  of  participants  proposed  that  test-item  5.2  (using  basic  spreadsheet  formula  to  assess  student 
grade),  test-item  11  (reflect/judge  credible  website)  and  test-item  17.4  (knowing  how  to  evaluate  credible 
webs)  were  the  hardest  test-item  with  EA  value  less  than  10%  of  total  participants.  This  finding  demonstrates 
that  this  group  of  participants  lack  the  skills  involving  the  use  of  a spreadsheet  application  and  also  the  skills 
and  knowledge  needed  that  would  enable  them  to  identify  and  judge  credible  websites.  These  skill  and 
knowledge  is  exceedingly  important  for  trainee  teachers  who  are  going  to  be  teaching  in  a Smart  School 
environment.  Definitely  we  do  not  want  the  teachers  to  be  teaching  the  school  children  with  resources  which 
are  unreliable  or  hearsay. 

Aside  from  that,  the  findings  also  indicated  that  the  learning  domain  tested  below  were  easy  for  this  group 
of  participants,  where  none  of  the  participants  scored  these  items  as  HA  or  HNA  (Table  2). 


Table  2.  Learning  domains  with  no  Harder  Achieved  or  Harder  Not  Achieved  score 


Item  no 

Learning  domain  tested 

1 

manage 

3 

basic  ICT  capabilities 

4 

basic  ICT  capabilities 

5.1 

assess 

10 

Internet  navigation  & search 

18.1 

perform  analysis  & producing  appropriate  document 

18.2 

perform  analysis  & producing  appropriate  document 

19 

basic  ICT  capabilities 

Eigure  2 also  showed  more  than  10%  of  the  participants  scored  ENA  on  five  test-items.  The  learning 
domains  were: 
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Table  3.  Learning  domains  with  Easier  Not  Achieve  score 


Item  no 

Learning  domain  tested 

3 

basic  ICT  capabilities 

6 

perform  analysis  & producing  appropriate  document 

7 

plan  & define 

8 

access 

9 

manage 

Rasch  model  suggested  that  these  learning  domains  (Table  3)  should  be  easy  to  the  participants.  However, 
more  than  10%  of  them  were  unable  to  correctly  complete  the  tasks.  The  reason  might  be  because  of 
carelessness,  or  might  even  be  because  of  their  over  confident  attitude  they  misread  the  task  requirement  and 
wrongly  executed  the  tasks.  It  might  also  be  because  the  already  had  the  knowledge  but  they  had  forgotten 
how  to  execute  them,  due  to  uncommon  use  of  the  task  in  the  participants’  daily  work.  Whatever  the  reasons, 
the  participants  needed  to  revisit  this  group  of  tasks  immediately  and  resolve  the  problem. 


7.  CONCLUSION 

Based  on  this  findings,  the  ICT-literacy  TBA  instrument  identified  that  this  group  of  participants  suggested 
that  the  assessing  information;  integrating  information  from  multiple  sources;  and  evaluating  the  information 
that  they  have,  as  the  hardest  ICT-literacy  skills.  These  skills  however,  were  among  the  most  important  ICT- 
based  skills  that  a teacher  must  have. 

The  ability  to  pin-point  to  the  exact  ICT-literacy  ability  that  the  trainee  teachers  lack  might  help  them  and 
their  institution  of  higher  learning  in  providing  necessary  intervention  or  additional  skills  to  these  trainee 
teachers.  Their  training  can  target  their  specific  area  of  weakness,  and  redundancy  in  training  information  can 
be  avoided.  This  in  turn  can  save  a lot  of  trainer’s  time  and  cost. 
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ABSTRACT 

This  paper  reports  on  a student-created  and  facilitated  peer-tutoring  activity  to  assist  first- year  students  in  preparing  for 
their  final  exam  in  an  introductory  information  technology  course.  Tutors  at  the  CIS  Sandbox,  a collaborative  learning  lab 
at  an  American  university,  offered  a series  of  “Crunch  Sessions”  to  their  peers.  This  paper  examines  how  these  sessions 
model  characteristics  of  peer  tutoring,  and  describes  how  social  media  and  Internet-based  collaboration  tools  support  a 
blended  learning  environment  where  an  in-person  event  is  aligned  with  a web  presence.  Interviews  with  student  tutors 
and  participants  suggest  that  the  peer  relationship  between  tutors  and  tutees  was  conducive  to  an  open  exchange  of 
questions  and  answers  among  participants,  and  that  the  sessions  contributed  to  student  learning  for  those  who  attended. 
Implications  for  designing  and  researching  online  peer  learning  activities  in  higher  education  contexts  are  discussed. 


KEYWORDS 

Peer  Tutoring,  Informal  Learning,  Collaborative  Learning  Community,  Learning  Environments,  ICTs 


1.  INTRODUCTION 

The  Computer  Information  Systems  Learning  and  Information  Technology  Sandbox  (CIS  Sandbox)  opened 
as  a collaborative  computer  lab  during  the  fall  2011  semester  at  Bentley  University,  a business  university  in 
the  United  States.  Staffed  by  24  undergraduate  and  graduate  student  assistants,  also  referred  to  as  tutors,  the 
facility  since  has  become  a valuable  campus  destination  for  learning  about  and  with  new  technology,  through 
the  in-person  peer  tutoring  services  and  online  resources  it  provides.  In  that  context,  peer  tutors  are  student 
tutors  who  have  taken  courses  in  which  they  tutor  other  students.  Most  of  the  tutors  receive  compensation 
through  federal  work-study  or  the  university’s  student  work  program  for  their  services.  Undergraduate  tutors 
work  approximately  7 hours  per  week;  graduate  tutors  work  approximately  20  hours  per  week. 

Results  from  the  initial  phase  of  this  research  (Frydenberg,  2013)  showed  that  students  from  IT  101 
(Introduction  to  Information  Technology  and  Computer  Concepts),  a course  required  of  all  first -year  students 
at  the  university,  were  consistently  the  largest  cohort  using  the  CIS  Sandbox  in  any  given  semester.  As  a 
result,  many  students  had  developed  relationships  with  some  of  the  tutors,  who  provided  assistance  on 
homework  assignments  and  clarification  of  course  concepts.  Tutors  recognized  an  opportunity  to  leverage 
students’  comfort  in  the  CIS-Sandbox  to  provide  them  with  an  informal  experience  to  prepare  for  their  final 
exam  (common  across  all  sections  of  IT  101),  and  in  turn,  improve  the  impact  of  peer  tutoring  services 
offered  by  the  CIS  Sandbox  on  student  learning  outcomes. 

As  part  of  ongoing  efforts  to  create  a collaborative  learning  community  in  the  CIS  Sandbox,  this  paper 
reports  on  a student-created  and  student-facilitated  peer-tutoring  activity  called  a Crunch  Session  to  assist 
first-year  students  in  preparing  for  their  IT  101  final  exam.  Many  students  who  visit  the  CIS  Sandbox  during 
the  semester  seek  tutoring  assistance  from  their  peers,  so  a student -driven  end-of-semester  review  session 
was  a natural  extension  of  the  collaborative  pedagogic  approach  that  is  a guiding  principle  of  the  CIS 
Sandbox  and  its  tutors.  Three  Crunch  Sessions  took  place  during  the  final  night  of  the  fall  2011  semester, 
when  students  begin  studying  for  final  exams.  With  14  sections  of  the  course  offered  to  approximately  691 
students,  and  a facility  with  a capacity  of  50  students  in  person,  the  tutors  and  the  author  discussed  ways  to 
create  a student -centered  learning  experience  that  relied  on  Internet -based  collaboration  tools  (ICTs)  in  order 
to  enable  students  to  attend  and  participate  in  person  and  online. 
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This  practitioner  study  reports  on  a second  phase  in  the  evolution  of  the  CIS  Sandbox,  describes  a peer- 
tutoring  activity,  presents  a model  for  its  implementation,  and  qualitative  evaluation  results  that  indicate  its 
effectiveness  in  promoting  student  learning. 


2.  SOCIAL  MEDIA  AND  PEER  TUTORING 

Peer  tutoring  and  other  aspects  of  peer  learning  have  shown  to  be  effective  model  for  learning  and 
engagement  (Jackson  & Bruegmann,  2009),  (Topping  K.,  2005).  Distinctions  among  styles  and 
implementations  of  peer  learning  exist  in  the  literature.  Peer  tutoring  is  an  approach  in  which  one  student 
instructs  another  on  material  in  which  the  first  is  an  expert  and  the  second  is  a novice  (Topping,  2005). 

Most  peer  educational  techniques  are  structured  around  carefully  defined  social  roles  and  procedures 
(Damon  & Phelps,  1989,  p.  10).  While  Damon  and  Phelps  recognized  this  was  the  case  for  primary  school 
learning  environments  in  an  age  where  Internet  access  in  classrooms  was  unlikely  and  online  social  networks 
were  not  invented  yet,  their  observation  that  peer  learning  has  a social  component  can  be  extended  to 
incorporating  social  media  and  collaboration  tools  in  higher  education  learning  spaces  as  is  evident  today. 
Previous  practitioner  studies  have  investigated  peer  tutoring  and  instruction  through  video  podcasting 
(Frydenberg  M.  , 2008),  blogs  (Davi,  Frydenberg,  & Gulati,  2007)  and  collaboration  tools  and  social  media 
(McLoughlin  & Lee,  2007),  (Rourke  & Anderson,  2002),  (Wheeler,  2009).  Approaches  to  peer  learning  in 
the  CIS  Sandbox  build  upon  these  techniques. 

“Peer  tutoring  emulates  the  traditional  teacher-student  relationship  in  which  one  party  transmits  expertise 
to  another.  The  teaching  party  knows  the  answers  and,  through  explanation  and  drill,  tries  to  communicate 
them  to  the  learning  party.”  (Damon  & Phelps,  p.  1 1) 

Peer  tutoring  is  beneficial  to  those  who  are  tutored  as  well  as  to  the  tutors  themselves.  (Topping  K.  J., 
1996)  Entrusting  tutors  to  work  with  students  “imparts  to  them  a message  that  is  likely  to  bolster  their 
enduring  confidence  in  themselves  and  interest  in  intellectual  achievement.”  (Damon  & Phelps,  p.  17) 
Jackson  and  Bruegmann  (2009)  found  that  knowledgeable  and  skilled  individuals  increase  the  skill  and 
knowledge  of  those  with  whom  they  interact. 

With  the  introduction  of  distance  education  and  online  communities,  software  for  managing  peer 
learning,  and  intelligent  tutoring  systems,  “...information  technology  has  begun  to  permeate  peer  learning” 
(Topping  K.  , 2005,  p.  642).  Research  in  higher  education  contexts  shows  that  contributing  to  learning 
communities  through  social  media  tools  such  as  collaborative  editing  is  becoming  “a  ‘new’  learner-centered 
pedagogy”  that  promotes  learner-driven  content  creation  and  collaborative  knowledge  building  (McLoughlin 
& Lee,  2008). 

Milne  (2007)  claims  that  from  the  ubiquity  of  social  media,  the  Internet,  and  other  technology 
developments  has  emerged  an  “interaction  age”  where  collaboration  is  expected,  and  both  formal  and 
informal  learning  spaces  must  be  designed  to  support  these  interactions.  Such  collaborative  peer  learning 
extends  outside  the  classroom  using  ICTs  to  enhance  dynamic  peer-peer  interactions  in  addition  to  peer-tutor 
interactions.  These  increased  personal  and  virtual  interactions  can  sustain  and  nurture  a learning  community 
(Nunes,  McPherson,  Firth,  & Gilchrist,  2002). 

Taking  these  prior  research  findings  forward,  the  integration  of  social  media  and  ICTs  in  a CIS  Sandbox- 
sponsored  Crunch  Session  prior  to  final  exam  week  would  contribute  to  refining  our  pedagogic  possibilities 
and  help  sustain  a collaborative  learning  community. 

Given  alternatives  in  peer  tutoring  techniques  and  approaches,  and  student  desires  to  be  active 
participants  in  their  own  learning,  these  questions  emerged  as  part  of  this  study  to  design,  implement  and 
evaluate  a student-facilitated  peer-learning  session: 

• Will  attending  a student-centered,  peer-tutor  facilitated  Crunch  Session  help  to  prepare  students  for 
their  final  exam? 

• How  can  the  use  of  Internet-based  collaboration  tools  create  a virtual  collaborative  learning  space  to 
support  an  in-person  peer-facilitated  learning  session? 

• How  will  tutors  and  attendees  react  to  such  a student-centered,  peer-facilitated  learning  session? 
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3.  CREATING  AN  IT  101  CRUNCH  SESSION  IN  THE  CIS  SANDBOX 

IT  101  (Introduction  to  Information  Technology  and  Computing  Systems)  is  a first-year  required  course  for 
all  students  at  Bentley  University,  in  Waltham,  MA.  The  course  introduces  students  to  technology  concepts 
such  as  using  the  World  Wide  Web,  hardware  and  software,  operating  systems,  storage,  creating  web  pages, 
wireless  networking,  multimedia,  and  problem  solving  skills  using  Excel.  Bentley  University  offered  14 
sections  of  IT  101  during  the  Fall  2012  semester  to  approximately  691  students,  as  evidenced  by  the  number 
of  final  exam  scores  on  record..  This  is  the  number  of  students  for  whom  a final  exam  score  is  on  record.  In 
addition  to  specific  assignments  created  by  their  instructors,  students  across  all  sections  complete  ten 
standard  assignments  during  the  semester  and  take  a common  final  exam. 

The  CIS  Sandbox  is  a computer  lab  aligned  with  an  online  presence.  Prior  research  focused  on  its 
physical  renovation  and  use  of  social  media  tools  to  create  an  open,  social,  and  collaborative  learning  space 
(Frydenberg  M.  , Aligning  Open,  Physical,  and  Virtual  Spaces  in  the  CIS  Sandbox,  2013).  The  facility 
features  several  collaborative  work  areas,  a lounge  area  with  soft  seating  for  informal  conversation,  a wall- 
mounted  Google  TV  and  gaming  equipment,  an  interactive  whiteboard,  a large  projector  to  facilitate  group- 
learning  activities,  and  bright  colors  painted  on  the  walls  that  create  a warm  atmosphere.  This  facility 
replaced  a traditional  computer  lab,  whose  layout  included  dozens  of  computers  flanked  around  the  perimeter 
of  a dark  room  in  a basement  corner  of  an  academic  classroom  building  (Frydenberg,  2013). 

During  a brainstorming  conversation  about  ways  the  CIS  Sandbox  tutors  might  assist  IT  101  students  in 
their  final  exam  preparation,  one  tutor  proposed  holding  three  one-hour  long,  back-to-back  Crunch  Sessions 
in  the  CIS  Sandbox.  Figure  1 shows  a Crunch  Session  process  in  progress.  Students  gather  at  collaborative 
stations  to  view  the  student  facilitator’s  screen  also  displayed  on  the  interactive  whiteboard,  on  each  monitor 
in  the  room,  and  on  some  student  laptops.  The  facilitators  projected  additional  images,  such  as  the  Crunch 
Session  logo,  on  the  back  wall,  to  create  an  immersive  experience. 


Figure  1.  An  IT  lOI  Crunch  Session  in  progress. 

In  an  interview  after  the  Crunch  Sessions  took  place,  a student  facilitator  and  organizer  reflected  on  his 
pedagogic  vision  for  this  event  that  builds  on  the  core  tutoring  services  of  the  CIS  Sandbox: 

“A  first  for  the  CIS  department,  and  most  likely  the  university,  this  session  would  be  run  entirely  by 
undergraduate  tutors  and  graduate  students  in  the  CIS  Sandbox.  Students  often  came  to  us  during  the 
semester  to  explain  concepts  that  they  do  not  quite  understand.  We  wanted  to  conduct  our  review  session 
for  the  final  exam  the  same  way.  We  want  the  session  to  be  entirely  question  driven  and  will  advertise  it 
as  such.  We  will  prepare  a series  of  documents  to  reference  when  answering  questions  as  well  as  a reserve 
bank  of  questions  in  case  the  session  goes  silent.” 

These  sentiments  form  the  basis  of  the  first  phase  (needs  analysis  and  goal-setting)  of  a three-phase  model 
for  designing,  implementing,  and  evaluating  a Crunch  Session  process  as  shown  in  Figure  2. 
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3 Phase  Practitioner  Research  Process  Model  To  Design,  Implement  & 
Evaluate  Peer-tutoring  Crunch-Sessions  In  The  CIS  Sandbox 


Needs 


Needs  Analysis 
& Goal  Setting 


What  do 
students  need? 

What  can  we  do? 

How  can  we 
build  on  our 
core  services? 


Creating  a Collaborative  Pedagogic  Context 


Social  Awareness  In-Person 

Registration  Peer  Tutoring 

Collaboration  Session 


Communication 
Participation 
Virtual  Presence 


PRE-EVENT 


AT-  EVENT 


Outcomes 


Outcomes 
& Artifacts 


Final  Exam 
Scores 


Student  and 
Tutor  Session 
Evaluations 


Social  Media 
Updates 


Perceptions  of 
process  and 
benefits  gained 


Figure  2.  A Model  for  Planning  Peer-Facilitated  Crunch  Sessions. 


Because  the  tutors  are  students  themselves,  they  know  what  types  of  review  opportunities  their  peers  will 
find  most  beneficial.  Marketing  and  publicizing  the  sessions  to  generate  interest  was  equally  as  important  as 
planning  their  content.  Promoting,  facilitating,  and  supporting  the  Crunch  Sessions  relied  on  a variety  of 
collaborative  and  social  networking  tools  to  engage  students  online  and  in  person,  before,  during,  and  after 
the  event.  Phase  2 (planning  and  implementation)  involved  determining  Internet-based  collaborative  and 
social  media  technologies  that  would  raise  social  awareness  about  the  event,  allow  students  to  register,  and 
collaborate  to  create  learning  artifacts  through  the  form  of  sharing  questions,  even  before  the  event  began. 
Phase  3 of  the  model  urges  students  and  their  tutors  to  reflect  on  outcomes  and  artifacts  of  learning  in  order 
to  determine  the  success  of  the  Crunch  Sessions.  After  the  sessions,  student  tutors  posted  additional  questions 
asked  at  the  session,  and  photos  of  attendees  on  the  CIS  Sandbox  blog  and  Facebook  page. 

The  use  of  social  media  and  Internet-based  collaboration  tools  in  peer  learning  situations  can  help  to 
define  social  roles  and  procedures  for  creating  and  sharing  knowledge.  The  use  of  social  media  technologies 
for  peer  tutoring  extends  Damon  and  Phelps  1980’s  observation  that  “such  interactions  encourage  ....  active 
involvement  in  reasoning,  problem  solving,  and  the  social  exchange  of  ideas”  (Damon  & Phelps,  p.  17)  into  a 
context  of  socially  connected  networked  learners.  Prior  to  the  event,  student  tutors  promoted  the  event  on  the 
CIS  Sandbox  blog,  and  on  Facebook.  The  blog  linked  to  an  events  page  on  Facebook,  where  students  could 
inform  their  peers  that  they  were  attending.  Eventbrite,  an  event  management  service,  allowed  students  to 
reigster,  and  managed  a waiting  list.  The  student  organizers  recognized  the  need  for  these  sessions  to  be 
student-driven,  and  made  these  intentions  clear  on  the  Crunch  Session  Eventbrite  page,  shown  in  Eigure  3(a), 
where  they  admonished  their  peers,  “No  Questions  = Lame  Session.”  When  students  registered,  the  Eacebook 
and  Eventbrite  sites  invited  them  to  submit  questions  on  Stixy.  Stixy  is  a popular  ICT  for  sharing  documents, 
photos,  links,  and  notes  with  a group.  Students  could  review  the  Stixy  board,  see  questions  posted  by  their 
peers,  add  comments,  or  post  their  own  questions  anonymously  before  the  session  started.  The  online  sticky 
board  created  a forum  for  students  who  might  otherwise  be  intimidated  about  asking  questions  in  person 
during  the  session.  This  also  allowed  the  student  facilitators  to  prepare  to  provide  review  materials  related  to 
these  questions.  Eigure  3(b)  shows  a section  of  the  Stixy  board  containing  student  questions. 
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TYPE  END  QUANTITY 

IT  101  Review  Session  Ended  Free  Sold  Out 


Event  Details 

Studying  for  the  IT  101  final  and  realizing  you  doni  remember  what  a VLOOKUP  Function  is?  Do  you 
need  help  understanding  what  style  to  use  to  make  your  web  page  Blue?  Are  your  3D  references  in 
excel  coming  out  2D?  Then  this  review  session  may  be  for  you! 


From  7:00-6:CC  PM  the  CIS  Sandbox  staff  will  be  hosting  a review  session  open  to  ALL  SECTIONS  of 
IT  101  to  help  answer  any  questions  you  might  have  before  your  final.  The  session  will  bs  held  In  the 
CIS  Sandbox  (Smith  234)  on  December  1Dth.  TTiere  is  a capacity  to  the  room  so  make  sure  you  sign 
you  and  your  friends  up  early! 


"PLEASE  NOTE"  THIS  IS  A QUESTION  DRIVEN  REVIEW  SESSION.  WE  WILL 
ANSWER  ANY  QUESTIONS  YOU  MIGHT  HAVE  ON  THE  MATERIAL  AND  WALK 
THROUGH  PROBLEMS  THAT  YOU  REQUEST.  No  Questions=Lame  Session. 


Submit  some  questions  ahead  of  time 
here:  http:/Afvww.stixy.comfguest/244581 

(a) 


(b> 


Figure  3.  (a)  Eventbrite  promotes  the  student-driven  session  and  encourages  students  to  submit  questions, 
(b)  Students  contribute  questions  on  Stixy  prior  to  the  Cmnch  Sessions. 


At  the  event,  participants  communicated  with  the  student  facilitators  via  simplemeet.me,  a browser-based 
chat  room.  The  student  facilitators  chose  this  tool  because  students  can  easily  connect  without  needing  to 
create  an  account  on  another  messaging  service.  The  student  facilitators  discussed  having  students  submit 
questions  and  comments  via  Twitter  using  a common  event  hashtag,  but  were  concerned  that  some  of  their 
peers  might  not  have  an  account  on  that  online  social  service. 

To  ensure  that  participants  could  see  the  presenter’s  screen  throughout  the  session,  the  student  organizers 
configured  join.me,  a free  screen-sharing  application,  to  share  the  presenter’s  screen  on  each  of  the  large 
screen  monitors,  interactive  whiteboards,  and  Google  TV  in  the  CIS  Sandbox.  They  also  provided  the  access 
code  for  students  to  be  able  to  view  the  presenter’s  screen  on  their  own  laptops,  tablets,  and  smart  phones. 

For  the  benefit  of  those  who  could  not  attend  in  person,  one  of  the  Crunch  Sessions  was  recorded  and 
streamed  over  the  Internet  using  UStream.TV,  a free  web-based  video  streaming  service. 


4.  PRELIMINARY  FINDINGS 

This  section  highlights  preliminary  findings  from  the  evaluation,  which  offer  an  initial  snapshot  of  the 
effectiveness  of  this  approach.  In  reflecting  on  qualitative  and  quantitative  data  related  to  the  IT  101  Crunch 
Sessions  and  CIS  Sandbox  usage  during  the  fall  2012  semester,  the  discussion  considers  the  following 
artifacts: 

• a voluntary  online  survey,  offered  at  the  end  of  the  semester,  to  all  IT  101  students  after  completing 
their  final  exam 

• attendance  information  obtained  from  student  “swipe-ins”  throughout  the  semester, 

• interviews  with  students  who  attended  the  review  sessions  and  tutors  who  planned  them 

• final  exam  grades  from  students  in  all  sections  of  the  course 

This  data  offers  a preliminary  understanding  of  student  experiences  at  the  CIS  Sandbox  and  during  the 
Crunch  Sessions.  Students  who  visit  the  CIS  Sandbox  swipe  in  with  their  ID  cards;  4082  swipes  from  1415 
unique  visitors  were  recorded  during  the  fall  2012  semester.  Based  on  forms  that  tutors  complete  logging 
their  tutoring  activities,  it  was  determined  that  approximately  half  of  the  tutoring  that  takes  place  is  with  first 
year  students  seeking  assistance  in  their  IT  101  classes. 
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4.1  Quantitative  Reflection 

The  following  discussion  summarizes  findings  from  student  surveys.  302  IT  101  students  (132  female,  170 
male)  completed  an  end-of-semester  survey  asking  about  their  use  of  the  CIS  Sandbox  during  the  Fall  2012 
semester.  Of  those,  200  students  came  to  the  CIS  Sandbox  seeking  help  with  IT  101.  82  students  said  they 
never  went  to  the  CIS  Sandbox,  citing  the  most  common  reason  that  “I  didn’t  need  help,  I knew  what  I was 
doing”  (61  students)  or  they  got  help  from  classmates  or  friends. 

116  of  the  students  who  responded  revealed  that  they  sought  help  on  their  Excel  exercises;  89  students 
sought  tutoring  on  their  web  assignment.  120  students  say  they  visit  once  or  twice  a semester;  56  students 
visit  at  least  monthly;  18  students  visit  at  least  weekly. 

Online  registrations  from  Eventbrite  showed  that  120  students  enrolled  in  IT  101  signed  up  to  attend  a 
Crunch  Session  in  person.  Usage  stats  showed  that  the  video  received  40  views.  Of  those  who  attended  the 
Crunch  Sessions,  all  students  who  responded  to  a survey  strongly  agreed  that  they  attended  because  they 
thought  doing  so  might  help  them  get  a better  grade  on  the  final  exam.  83%  said  they  brought  their  computers 
to  work  through  the  problems  that  the  peer  tutors  presented. 

A first  examination  of  final  exam  scores  of  students  who  attended  the  Crunch  Sessions  compared  to  those 
who  did  not  shows  that  students  who  attended  had  a slightly  higher  average  and  median  score  over  those  who 
did  not  attend.  75.68%  of  session  attendees  scored  in  the  80%  to  100%  grade  range  on  the  final  exam, 
compared  with  66.23%  of  those  who  did  not  attend,  whose  final  exam  scores  were  in  that  grade  range. 
67.25%  of  all  students  who  took  the  exam  scored  80%  or  higher.  It  is  important  to  note  this  difference  is  not 
statistically  significant  for  a variety  of  reasons:  Students  self-selected  to  attend  or  not  to  attend  the  Crunch 
Sessions  based  on  their  subjective  perceptions  of  whether  or  not  attending  would  be  useful  to  them.  Some  did 
not  attend  because  they  felt  they  were  good  students  and  did  not  need  additional  help.  Others  who  considered 
themselves  “good”  students  may  have  chosen  to  attend  for  more  social  than  academic  reasons.  Einally,  the 
capacity  of  the  physical  space  in  the  CIS  Sandbox  could  not  accommodate  all  students  to  attend  in  person, 
even  if  they  wanted  to,  creating  unequal  review  experiences  for  various  groups. 

4.2  Qualitative  Reflection 

Student  tutors  and  student  participants  participated  in  interviews  regarding  their  experiences  as  facilitators 
and  attendees  of  the  Crunch  Sessions  shortly  after  the  sessions  were  held.  One  tutor  described  how  he 
prepared  to  facilitate  a Crunch  Session,  citing  how  he  researched  the  most  commonly  asked  questions  from 
reports  of  peer  tutoring  sessions  in  the  CIS  Sandbox  that  tutors  complete  throughout  the  semester: 

“In  preparation  for  the  session,  we  carefully  observed  trends  in  tutor  reports  that  were  submitted 
throughout  the  year  to  see  what  concepts  most  students  were  coming  in  for.  In  the  case  of  Excel,  we 
would  start  with  explaining  the  concept  of  a VLookup  function,  which  often  was  difficult  for  students  to 
grasp.  We  reviewed  the  example  that  they  were  used  to  from  their  classroom  experience  and  also  created 
an  entirely  new  example  and  walked  through  it  until  there  were  no  more  questions.” 

Rourke  and  Anderson  (2002)  have  identified  roles  for  establishing  a teaching  presence  in  peer -led  online 
discussions,  including  instructional  design,  facilitating  discourse,  and  direct  instruction.  Comments  from  a 
student  facilitator  sharing  his  sense  of  the  room  as  the  sessions  began  recognized  these  roles: 

“The  room  was  packed  and  it  definitely  had  the  feeling  of  an  informal  learning  session.  With  the  chat 
widget  on  the  white  board  and  not  a professor  in  sight,  students  were  relaxed  and  more  prone  to  ask 
questions  that  they  might  have  thought  someone  would  ridicule  them  for  in  a classroom  environment. 
Often  times  you  could  see  students  move  forward  in  their  seats  during  certain  questions  as  if  they  were 
waiting  for  someone  else  to  ask  it,  and  not  a single  student  snickered  or  laughed;  there  were  no  wrong 
questions  here.  Everyone  was  in  the  room  to  learn  from  students  who  were  currently  in  the  class  or  who 
had  already  taken  the  class. 

The  sessions  all  began  awkwardly,  with  moments  of  silence.  Because  we  were  students  ourselves  we 
knew  what  it  was  like  to  be  at  a review  session,  not  wanting  to  appear  ‘stupid’  for  asking  a question  that 
was  probably  covered  in  class...  After  the  first  example,  however,  students  were  much  more  apt  to  ask  a 
question  or  look  for  a clarification.  The  remainder  of  the  session  ran  quite  smoothly  until  there  were  no 
more  questions.  Usually  as  time  went  on  students  became  more  focused  and  attentive.  As  students  see  the 
value  that  their  tutors  bring,  they  are  much  more  likely  to  treat  us  as  a resource  as  well  as  a peer,  unlike 
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the  student-professor  relationship.  Also,  because  there  is  the  mentality  that  ‘we’re  all  in  this  together,’ 
students  felt  like  they  were  being  fed  secret  details  or  inside  information  because  they  were  learning  from 
other  students  like  themselves.” 

Finally,  tutors  reflected  on  how  they  might  know  if  the  sessions  were  successful:  “We  weren’t  looking  to 
teach  new  concepts  or  to  make  students  unlearn  something  and  relearn  it  a different  way.  We  were  looking  to 
clarify  areas  where  students  had  a small  gap  in  their  knowledge,  or  needed  something  explained  again.  There 
were  very  few  questions  at  the  conclusion  of  the  event.  We  saw  students  work  through  problems  and  obtain 
the  correct  answers.  We  were  satisfied.” 


5.  DISCUSSION  AND  NEXT  STEPS 

ICTs  played  a role  in  all  aspects  of  the  Crunch  Sessions.  The  Crunch  Sessions’  success  relied  on  motivated 
and  tech-savvy  tutors  who  were  able  to  introduce  the  use  of  ICTs  to  facilitate  interaction  among  participants 
before,  during,  and  after  the  sessions. 

Prior  to  the  in-person  session,  students  relied  on  social  networking  applications  to  get  information  and 
sign  up.  They  prepared  by  taking  online  quizzes,  reviewing  resources  on  the  CIS  Sandbox  blog,  and 
submitting  questions  anonymously  to  an  online  sticky  board. 

During  the  sessions,  to  accommodate  a large  number  of  students  in  a small  space,  students  entered  their 
questions  in  the  chat  window  as  demos  took  place.  Presenters  returned  to  those  questions  to  ensure  they 
answered  them  adequately.  The  chat  window  served  as  a “parking  space”  for  students  to  ask  questions  as 
they  thought  of  them,  without  disrupting  or  interrupting  the  demonstration  in  progress.  The  use  of  web-based 
communication  tools  helped  to  create  an  in-person  environment  that  was  immersive,  inviting,  and  conducive 
to  learning. 

After  the  sessions,  a video  posted  online  allowed  students  who  wanted  additional  review,  or  who  were 
unable  to  attend  in  person,  the  opportunity  to  learn  at  their  own  pace. 

While  designed  for  technology-based  courses,  the  Crunch  Session  approach  extends  to  other  areas  such  as 
mathematics,  where  problem  solving  in  small  groups  is  beneficial.  Screen  sharing  technology  is  beneficial  in 
courses  that  rely  on  students  gaining  proficiency  in  software  applications,  while  the  use  of  chat,  and  ICTs  for 
promoting  the  sessions  and  gathering  questions  ahead  of  time  are  applicable  across  several  disciplines. 

The  success  of  the  first  IT  101  Crunch  Session  inspired  other  tutors  to  offer  similar  review  sessions  to 
their  peers  for  additional  courses  in  future  semesters.  After  hearing  about  the  sessions,  CIS  instructors 
weighed  in  on  ways  to  improve  them,  and  tutoring  in  general,  in  order  to  promote  critical  thinking  and 
problem  solving.  One  instructor  commented  on  the  potential  value  of  frequent  review  sessions: 

“I’ve  heard  about  the  ITIOI  review  session  and  I’m  very  happy  for  student-led  efforts  in  learning.  We 
have  to  be  careful  though  not  to  reduce  it  to  ‘here’s  how  you  do  your  next  homework’  type  sessions. 
There  are  several  practice  problems  that  we  do  in  class,  sometimes  quite  a few  students  don’t  manage  to 
finish  those...  What  if  the  [tutors]  ran  regular  sessions  helping  students  either  finish  the  practice  problems 
from  class,  or  even  doing  another  practice  problem?” 


6.  CONCLUSION 

Students  found  the  format,  timing,  and  ICTs  used  in  the  Crunch  Sessions  to  be  helpful  and  engaging,  and  that 
the  use  of  social  media  and  Internet-based  collaboration  tools  contributed  to  the  learning  experience.  These 
students’  remarks  were  typical:  “the  IT  101  Crunch  Sessions  really  helped.  I liked  the  idea  of  chatting,  where 
people  could  ask  questions  to  assistants.”  Said  another  student,  “The  review  session  was  really  helpful  and 
went  over  the  things  I didn't  know  how  to  do.” 

The  IT  101  Crunch  Sessions  met  an  identified  student  need,  and  aligned  a three-phase  participatory 
research  process  model  with  the  goal  of  building  a collaborative  learning  community  in  the  CIS  Sandbox  by 
involving  students  as  co-designers  of  pedagogic  practices.  Students  worked  to  create  a peer-facilitated 
learning  environment  that  extends  their  roles  as  peer  tutors.  Based  on  these  encouraging  early  results,  it 
appears  that  the  model  works.  Further  research  will  be  necessary  to  ascertain  the  impact  of  peer  tutoring 
combined  with  the  use  of  ICTs  at  future  Crunch  Sessions  for  information  technology  students. 
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ABSTRACT 

During  the  years  2008-2013  Ministry  of  Education  in  the  Republic  of  Slovenia  prepared  largest  project  of  informatization 
of  educational  institutions.  One  of  the  main  goals  of  the  project  E-Education  is  the  development  of  the  standard 
E-Competent  Teacher,  School  Principal  and  IT  Expert.  The  standard  E-Competent  Teacher,  Principal  and  IT  Expert 
defines  the  key  e-competencies  that  every  teacher,  pre-school  teacher  or  teacher  in  a student  residence  needs  in  order  to 
work  with  their  students;  every  school  principal  or  deputy  school  principal  needs  in  order  to  manage  their  school;  and 
every  IT  expert  or  IT  co-ordinator  needs  in  order  to  successfully  collaborate  in  planning  school  work  and  classes  by  using 
ICT.  The  second  part  shows  the  individual  indicators  of  the  e-Education  project. 


KEYWORDS 

e-Education,  e-competence  way,  informatization  of  educational  institutions,  didactic  advice. 


1.  THE  OBJECTIVES  OE  THE  E-EDUCATION  PROJECT 

In  2007,  the  government  of  the  Republie  of  Slovenia  approved  the  Strategy  for  Development  of  Information 
Society  in  the  Republic  of  Slovenia  - si2010,  which  defined  the  national  framework  for  the  development  of 
information  society  in  Slovenia.  It  states:  "By  2013,  establish  an  effective  and  fully  computer-supported 
national  education  system  and  also  by  this  ensure  sustainable  economic  growth,  prosperity  and  quality  of  life 
of  all  citizens  of  the  Republic  of  Slovenia.  At  the  same  time  become  synonymous  with  one  of  the  most 
successful  societies  in  the  world  that  are  knowledge -based,  constant  innovation  and  rapid  development.  To 
ensure  inseparable  connection  between  all  processes  and  content  with  feeding  and  gaining  knowledge  from 
information  and  communication  technology;  there  should  be  no  curriculum,  subject  and  participant  where 
this  would  not  be  present.  " 

The  strategy  includes  providing  suitable  equipment  for  schools,  developing  e-content  and  training 
education  professionals  (Kreuh  and  Brecko,  2011).  The  project  E-Education  is  based  on  all  of  these  aims 
because  the  only  way  to  build  a 21st  century  school  that  can  be  characterized  as  e-competent  is  to  create  an 
e-leaming  environment,  to  develop  suitable  e-content,  and  - most  importantly  - to  educate  e-competent 
teachers  who  can  use  this  content  in  a suitable  e-leaming  environment. 

Ministry  of  Education  and  Sports  is  in  response  to  the  challenges  of  teaching  and  learning  in  the 
information  society  offered  several  strategic  focused  projects.  One  of  the  key  projects  is  certainly  the  project 
"e-Education",  which  is  co-ordinated  within  the  web  portal  (e -center)  and  consists  of  three  sets  (three 
projects),  as  follows:  consultancy  development  and  implementation  and  support  to  e-Competence  schools, 
development  and  implementation  of  e-leaming  materials,  development  and  implementation  of  training  for 
e-competent  teachers. 

The  main  objective  that  we  pursue  in  the  course  of  the  project  "Consultancy  development  and 
implementation  and  support  to  e-Competence  schools"  ("e-support"),  is  a continuation  and  acceleration  of  the 
process  of  computerization  of  each  educational  institution,  depending  on  their  current  status  in  management, 
teaching  and  technical  fields.  We're  talking  about  counselling  (for  school  management,  teachers,  school 
professionals),  educational  support  in  teaching  and  learning  (for  teachers),  technical  assistance  to  educational 
institutions. 


279 


ISBN:  978-972-8939-88-5  © 2013  lADIS 


The  main  goal  we  pursue  in  the  eourse  of  the  projeet  "Development  and  delivery  of  training 
eourses/seminars  for  e-eompetent  teaehers"  is  the  development  of  proposals  for  standard  e-eompetent  sehool, 
e-eompetent  teaeher,  prineipal  and  IT  expert.  In  these  eontexts  we  prepare  and  implement  training  and  other 
aetivities. 

Our  eollaborators  in  projeet  are  teaehers,  prineipals,  experts  from  university  departments  and  other 
institutions.  We  are  establishing  eloser  eooperation  with  eertain  university  departments  and  other  institutions 
in  the  field  of  edueation  both  at  home  and  abroad.  It  ean  be  said  that  the  projeet  "e-Edueation"  provides  an 
opportunity  for  all  edueational  institutions,  to  foeus  on  the  use  of  new  teehnologies,  beeome  more  innovative 
and  effieient  in  their  work  with  students,  praetitioners  and  professional  assoeiates.  By  training  at  various 
seminars  and  eounsellings,  eondueted  both  live  and  on-line,  professionals  aequire  new  eompetenees  in 
teaehing,  whieh  they  ean  test  in  the  elassroom  already  during  the  seminar. 

One  of  the  main  goals  of  the  projeet  E-Edueation  is  the  development  of  the  standard  E-Competent 
Teaeher,  Sehool  Prineipal  and  IT  Expert.  They  are  the  most  important  faetors  in  establishing  e-eompetent 
sehools.  The  standard  E-Competent  Teaeher,  Prineipal  and  IT  Expert  defines  the  key  e-eompeteneies  that 
every  teaeher,  pre-sehool  teaeher  or  teaeher  in  a student  residenee  needs  in  order  to  work  with  their  students; 
every  sehool  prineipal  or  deputy  sehool  prineipal  needs  in  order  to  manage  their  sehool;  and  every  IT  expert 
or  IT  eo-ordinator  needs  in  order  to  sueeessfully  eollaborate  in  planning  sehool  work  and  elasses  by  using 
ICT.  E-eompetent  sehool  so  ineludes:  e-eompetent  professionals,  modem  developed  e-eontent  (e-leaming 
materials)  and  e-leaming  environment. 

To  aehieve  this  objeetive,  the  e-Edueation  projeet  developed  the  following  eontent:  E-eompetenees  (for 
teaehers,  prineipals,  IT  experts).  Seminars  (live  and  on-line  in  virtual  elassrooms).  Counselling  (experts 
advising  to  a small  group  of  3-5  teaehers,  advantage  is  the  flexibility  and  adaptability  of  eontent),  Teehnieal 
support,  Edueation  of  teaeher  trainers  (three  times  annually),  Slovenian  Edueational  Network  SIO  - web 
portal  www.sio.si  (with  edueation  eatalogue,  e-eommunities,  e-leaming  materials,  information),  e-leaming 
materials,  e-textbooks.  International  eonferenee  SIRIKT,  Information  system  to  support  projeet.  Evaluation, 
self-evaluation. 

Edueational  institutions  have  deeided  freely  if  they  would  like  to  be  ineluded  in  the  e-Edueation  projeet. 
Being  a part  of  the  projeet  gave  them  aeeess  to  edueation,  eontent  and  serviees. 


2.  HOW  TO  BUILD  A 21ST  CENTURY  SCHOOL? 

Beeause  of  the  rapid  development  of  digital  teehnologies  individuals  require  an  inereasing  number  of  various 
skills,  espeeially  teehnieal,  eognitive  and  soeial  skills,  in  order  to  perform  tasks  and  solve  problems  in  digital 
environments  of  their  personal  and  professional  lives  (Eshet-Alkalai  2004).  This  also  means  more 
responsibility  for  teaehers  and  other  edueation  professionals,  sinee  in  the  e-permeated  world,  ‘those  who  ean 
understand  and  eomfortably  use  e-faeilities  are  signifieantly  advantaged,  in  terms  of  edueational  sueeess, 
employment  prospeets  and  other  aspeets  of  life’  (Martin  2005). 

The  aim  of  modem  innovative  praetiee  is  development  of  teaehing  methods  that  will  improve  learning 
outeomes  for  students  and  better  prepare  them  to  live  and  work  in  the  21st  eentury  (Vuorikari,  Garoia, 
Balanskat,  2011).  To  sueeeed  in  today's  soeiety,  rieh  with  information  and  knowledge,  students  and  teaehers 
must  use  teehnology  effeetively.  The  introduetion  of  ICT  in  the  elassroom  affeets  in  following  ways:  It  helps 
students  to  develop  the  skills  neeessary  to  sueeessfully  live  and  work  in  the  21st  eentury,  to  eneourage 
teaehers  to  improve  the  way  of  teaehing  in  the  elassroom  through  interaetive  and  dynamie  resourees  offered 
by  ICT,  and  provides  more  motivation  and  a rieher  learning  experienee  for  students  (Brecko,  Vehovar,  2008). 

A foundation  for  this  is  eourse  syllabi  that  are  based  on  lifelong  eompeteneies.  One  of  these  eompeteneies 
is  the  development  of  learners’  digital  literaey.  Sehools  are  eommitted  to  use  modem  teehnology  in  their 
elassrooms,  but  only  by  training  of  all  sehool  employees  they  will  be  able  to  fulfd  other  requirements  set 
forth  in  the  syllabi.  E-eompetent  prineipals,  teaehers  and  IT  experts  are  the  most  important  faetors  in 
establishing  e-eompetent  sehools. 
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3.  E-COMPETENCIES  - WHY  AND  HOW? 

The  standard  stems  from  many  years  of  work  focused  on  the  role  of  ICT  in  education.  It  takes  into  account 
the  findings  of  numerous  ICT  studies  conducted  in  Slovenia  and  abroad,  the  knowledge  acquired  in  the  many 
years  of  training  education  professionals,  the  development  of  e-materials,  and  the  efforts  of  the  state  to 
provide  suitable  equipment  for  schools.  In  addition,  in  the  process  of  devising  the  standard,  a number  of 
non-Slovene  standards,  licenses  and  strategies  in  the  field  of  ICT  were  examined.  The  way  to  the  new 
standard  is  based  on  6 key  competencies  which  define  all  the  competencies  and  e-competencies  that  help 
teachers,  pre-school  teachers,  principals  and  ICT  co-ordinators  achieve  digital  literacy.  Of  course,  they  are 
related  to  the  requirements  for  digital  literacy  of  students  specified  in  the  course  syllabi.  Teachers  and 
pre-school  teachers  should  be  trained  to  carry  out  the  aims  specified  in  these  documents. 

Every  education  professional  can  acquire  e-competencies  as  authentically  as  possible:  the  content  and 
examples  of  good  practices  are  selected  based  on  the  subject  or  subject  area  relevant  for  the  institution  and 
thus  directly  transferable  to  practice;  take  part  in  “blended  learning”  (live  and  on-line)  seminars  in 
e-classrooms,  which  makes  it  possible  for  the  participants  to  learn  experientially  and  work  on  collaborative 
projects;  Undergo  competencies  assessment  in  all  seminars. 

In  2013  we  introduce  independent  competencies  assessment,  which  will  enable  the  already  skilled 
teachers,  pre-school  teachers,  school  principals  and  ICT  co-ordinators  to  assess  their  e-competencies  or  skills 
independently,  without  taking  part  in  training  seminars.  We  also  offer  consultancy  services  which  are 
intended  to  motivate  education  professionals  to  use  ICT  in  their  everyday  work  and  to  help  them  choose  a 
suitable  training  seminar.  Post-seminar  consultancy  can  further  develop  and  perfect  the  acquired 
competencies.  Seminar  participants  can  also  collaborate  in  e-communities  where  they  exchange  their 
experiences  and  learn  about  new  developments  in  teaching. 

3.1  Three  Levels  of  Digital  Literacy 

Following  the  needs  of  today’s  society  and  the  development  and  needs  of  learners,  the  project  E-Education 
defines  and  describes  the  digital  competencies  or  e-competencies  that  represent  the  basic  level  of  digital 
literacy.  They  refer  to  teachers,  pre-school  teachers,  ICT  co-ordinators,  school  principals  and  deputy  school 
principals. 

Our  seminars  mostly  focus  on  the  first  level  of  digital  literacy  since  they  support  the  development  of 
digital  competencies.  Partly,  the  second  level  is  also  included  because  the  blended  learning  seminars  enable 
participants  to  put  into  practice  the  knowledge  they  gain  on  various  strategies,  approaches,  content  and 
materials.  By  doing  so,  they  apply  the  acquired  competencies  to  their  professional  field.  However,  it  depends 
on  each  individual  to  what  extent  they  use  the  new  approaches  in  their  work  after  the  training,  and  only 
eventually  (probably  after  years  of  work)  can  they  reach  the  third  level,  i.e.  the  level  of  digital  transformation 
that  includes  innovation  and  creativity  in  the  use  of  ICT. 

3.2  Personal  Training  Plan 

The  focus  of  the  seminars  on  one  key  e-competency  allows  teachers,  pre-school  teachers  and  school 
principals  to  plan  their  way  towards  e-competency.  Each  individual  should  therefore  prepare  a personal 
training  plan  prior  to  the  training.  One  of  the  tasks  in  the  entrance  seminar  Collaboration  in  the  On-line 
Learning  Environment  focuses  on  this. 

Teachers  should  use  the  Slovene  Educational  Network  web  portal  to  choose  suitable  seminars  from  the 
Seminar  Catalogue.  Their  search  can  be  based  on  the  subject  area  (for  example,  languages,  physics, 
kindergartens,  school  principals  ...)  or  on  the  main  e-competency  developed  by  the  seminar.  For  every 
seminar  there  is  a description  of  content  and  aims,  so  the  potential  participants  can  also  consider  the  subject 
matter  they  might  be  interested  in.  Personal  training  plans  should  be  part  of  Institutional  Informatization 
Plans,  which  are  prepared  by  educational  institutions  and  are  consequently  part  of  their  E-Card. 
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3.3  Six  Key  e-competencies 


The  concept  of  reaching  e-competency  is  defined  in  the  document  The  Guidelines  of  the  Standard 
E-Competent  Teacher,  School  Principal,  IT  Expert  (Kreuh  and  Brecko  2011)  and  based  on  the 
implementation  of  six  key  e-competencies  (Figure  1). 
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Figure  1 . Key  e-competencies 


3.4  On  the  Way  towards  e-Competent  Schools  through  Seminars  - One 
Seminar,  One  Key  Competency 


Our  way  towards  e-competency  is  based  on  seminars  that  follow  the  principle  ‘one  seminar,  one  key 
competency’.  Each  seminar  focuses  on  developing  one  of  the  six  key  competencies.  The  key  competencies 
are  marked  with  numbers  1 to  6 (Figure  2). 

The  seminars  for  teachers,  principals  and  IT  experts  or  ICT  co-ordinators  are  devised  by  teachers, 
members  of  various  subject  groups  or  subject  area  groups,  and  aimed  at  various  target  groups  in  education. 
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Figure  2.  Key  e-competencies  marked  from  Cl  to  C6 

3.4.1  Precondition:  basic  ICT  skills 

A precondition  for  achieving  digital  literacy  and  being  on  your  way  towards  e-competency  is  basic  ICT 
skills.  They  encompass  six  areas  that  include  working  with  texts,  spreadsheets  and  presentations,  the  use  of 
the  web  and  e-mail,  as  well  as  the  use  of  the  necessary  hardware  and  software.  By  responding  to  our 
questionnaire,  teachers,  principals,  IT  experts  or  ICT  co-ordinators  can  conduct  a self-assessment  of  their 
basic  ICT  skills.  Achieving  basic  level  of  ICT  knowledge  and  skills  provides  participants  smooth  work  in 
seminars. 
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3.4.2  The  way  from  Cl  to  C6 

Entrance  seminar  for  professional  staff 

The  first  step  on  the  way  towards  a teaeher’s  e-eompeteney  is  the  entranee  seminar  Collaboration  in  the 
On-line  Learning  Environment.  The  aim  of  the  seminar  is  to  establish  both  live  and  distanee  eollaboration. 
By  using  a wide  range  of  tasks,  teaehers  eollaborate,  eommunieate  and  learn  about  personal  data  safety  in  the 
virtual  environment. 

In  their  seeond  step  on  the  way  towards  e-eompeteney,  teaehers  seleet  and  eomplete  at  least  four 
seminars.  They  have  to  take  into  eonsideration  that  the  seminars  should  be  ehosen  so  that  they  inelude  the 
first,  seeond,  third  and  the  fifth  eompeteneies.  Teaehers  ean  seleet  the  seminars  from  the  eatalogue  available 
on  the  web  portal  www.sio.si.  The  following  options  are  available:  Seminars  in  their  own  subjeet  area, 
seminars  from  other  subjeet  areas  (for  example,  a language  teaeher  ean  ehoose  a seminar  primarily  meant  for 
primary  teaehers  or  art  edueation  teaehers),  seminars  for  professional  staff  in  general. 

The  range  of  seminars  is  sueh  that  it  allows,  for  example,  a seeondary  sehool  speeialist  subjeet  teaeher  to 
reaeh  the  standard  despite  there  being  no  seminars  foeusing  on  their  speeifie  area. 

Choosing  seminars  that  foeus  on  other  subjeet  areas  provides  possibilities  for  interdiseiplinary 
eollaboration  and  opportunities  for  the  partieipants  to  learn  from  eaeh  other  even  though/beeause  they 
speeialise  in  different  subjeet  areas.  The  seminars  seleeted  by  teaehers  on  their  way  towards  e-eompeteney 
target  four  key  eompeteneies  that  are  marked  as  Cl,  C2,  C3  and  C5. 

The  eatalogue  on  the  web  portal  www.sio.si  allows  teaehers  to  seareh  for  seminars  based  on  key 
eompeteneies  or  based  on  subjeet  areas,  making  it  possible  for  them  to  easily  ereate  their  personal  training 
plans. 

After  sueeessfully  eompleting  the  seminars  that  they  seleet  on  their  way  towards  e-eompeteney  and  after 
aequiring  the  eompeteneies  Cl,  C2,  C3  and  C5,  teaehers  automatieally  aequire  the  eompeteney  C4  (Safe  use 
of  the  web,  ethieal  and  legal  use  of  information)  and  the  eompeteney  C6  (Plan,  perform,  evaluate  learning 
and  teaehing  by  using  ICT). 

3.5  A Principal  and  an  IT  Expert  or  ICT  Co-Ordinator  on  their  Way  towards 
e-Competency 

In  this  seetion,  a training  strategy  similar  to  the  one  for  teaehers  is  presented  for  prineipals  and  IT  experts  or 
ICT  eo-ordinators. 

A prineipal  on  their  way  towards  e-eompeteney  eompletes  four  seminars  targeted  at  prineipals.  Every 
seminar  has  a key  eompeteney:  the  seminar  On-line  Communieation  in  the  Working  Environment  and 
Elsewhere  foeuses  mainly  on  the  eompeteney  C2,  the  seminar  An  Appropriate  and  Critieal  Use  of  Data  and 
Information  in  Edueational  Institutions  on  C3,  the  seminar  Publishing  Data  on  C5,  and  the  seminar 
Prineipal’s  Planning  and  Monitoring  Work  by  Using  ICT  on  C6. 


Figure  3.  A diagram  presenting  the  way  to  become  an  e-competent  principal 

To  fulfil  the  aims  of  these  seminars  and  to  aequire  new  knowledge,  eonsultaney  is  also  available.  By 
working  on  seminar  tasks,  prineipals  identify  the  skills  that  they  require  to  run  their  institutions  and  ehoose 
the  type  of  eonsultaney  that  best  fits  their  needs.  Sueh  eonsultaney  is  eondueted  in  small  groups  and  foeused 
on  prineipals’  work. 

It  is  very  important  that  prineipals  on  their  way  towards  e-eompeteney  establish  mutual  eo-operation, 
whieh  takes  plaee  during  live  and  distanee  seminars  as  well  in  the  form  of  eommunieation  and  exehange  of 
good  leadership  praetiees  among  edueational  institutions. 
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An  IT  expert  or  ICT  eo-ordinator  on  their  way  towards  e-eompeteney  eompletes  two  eompulsory 
seminars  for  ICT  eollaboration  and  at  least  two  seminars  that  they  seleet  depending  on  the  subjeet  they  teach. 
The  two  compulsory  ICT  collaboration  seminars  focus  on  the  key  competencies  K2  and  K6,  while  the 
seminars  emphasizing  the  competencies  K5  and  K3  can  be  selected  freely. 

IT  experts  or  ICT  co-ordinators  follow  the  goal  of  acquiring  the  skills  that  will  make  it  possible  for  them 
to  support  the  ICT  needs  of  the  professionals  at  their  institutions.  They  will  co-operate: 

• With  teachers  to  plan  ICT  supported  lessons, 

• With  principals  to  plan  the  work  at  their  institution  and  introduce  changes. 

The  competencies  Cl  (Knowledge  and  awareness  of  ICT,  critical  use)  and  C4  (Safe  use  of  the  web, 
ethical  and  legal  use  of  information)  are  included  in  all  seminars  for  principals  and  IT  experts. 

As  the  nature  of  their  work  requires  a number  of  technical  skills,  IT  experts  and  ICT  co-ordinators  will  be 
offered  additional  training  in  this  field  as  well. 

3.6  Independent  Assessment  of  Competencies 

We  are  developing  ways  to  independently  assess  competencies,  so  that  the  skilled  self-leamers  will  be  able  to 
evaluate  their  competencies  and  attest  some  of  their  e-competencies  without  taking  part  in  the  seminars.  This 
means  that  they  will  achieve  e-competency  on  their  own,  using  distance  learning.  Independent  assessment 
will  be  regarded  as  equal  to  assessment  at  seminars.  However,  since  one  of  the  basic  requirements  of  the 
standard  E-Competent  Teacher  is  collaborative  work,  this  type  of  assessment  will  not  be  available  at  any 
time,  neither  will  it  be  completely  independent.  Candidates  wishing  to  prove  their  competency  in  this  way 
will  be  placed  in  groups  to  collaborate  by  following  the  instructions  of  the  group  administrator. 

3.7  E-card 

To  monitor  their  way  towards  e-competency,  teachers,  principals  and  IT  experts  will  be  issued  E-Cards.  The 
E-Card  will  enable  them  to  systematically  follow  the  progress  they  make  while  attending  the  training 
seminars  focusing  on  the  six  competencies.  When  the  six  competencies  are  acquired,  the  way  towards 
becoming  an  e-competent  teacher,  principal  and  IT  expert  in  the  project  E-Education  is  complete.  The 
E-Card  will  also  contain  information  on  the  role  of  E-Education  collaborators  and  participation  at  training 
sessions.  The  E-Card  represents  a list  of  skills  that  belongs  in  the  portfolio  of  every  education  professional. 
The  latter  is  increasingly  important  since  it  is  a record  and  a plan  of  every  individual’s  development. 

3.8  E-competent  School 

A contemporary  school  of  the  21st  century  or  an  e-competent  school,  as  such  school  is  named  within  the 
E-Education  project,  is  a school  of  e-competent  teachers  who  sensibly  use  e-content  in  e-leaming 
environments  and  follow  contemporary  teaching  and  learning  trends  centred  on  e-competent  children  and 
adolescents. 
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Figure  4.  An  e-competent  school  scheme 
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4.  THE  RESULTS  OE  THE  E-EDUCATION-AN  OVERVIEW  OF 
INDICATORS 

4.1  Number  of  New/Updated  and  Already  Carried  Out  Seminars 

There  were  173  new/updated  seminars  in  the  period  from  March  2009  to  the  end  of  2012.  All  of  them  have 
edited  and  regulated  virtual  classrooms  with  tasks  and  exercises  that  include  verification  of  skills  and 
competences.  Each  of  them  includes  tasks  of  online  safety.  In  the  period  from  March  2009  until  the  end  of 
December  2012  there  were  2088  seminars  carried  out. 

4.2  Number  of  Participants 


The  graph  below  shows  that  in  the  period  from  March  2009  to  the  end  of  December  2012,  we  planned  26,642 
participants  on  conducted  seminars,  in  fact,  the  seminars  attended  28,237  participants. 
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Graph  1 . Number  of  participants  in  seminars 

There  are  34,400  teachers,  educators  and  other  professionals  of  education  (on  pre-school,  primary  and 
secondary  school  level)  responsible  for  education  in  Slovenia  (Statistical  Office,  2012).  36,495  teachers  were 
included  in  counselling.  So  in  average  every  teacher  was  involved  in  counselling  of  e-Education  project.  It 
must  be  taken  into  consideration,  that  some  teachers  attended  number  of  different  counselling.  28,237 
teachers  were  participating  at  seminars.  A total  number  of  64,732  teachers  were  involved  in  teacher  trainings 
of  e-Education  project.  The  project  involves  680  trained  advisors  who  attended  4901  training  courses  for 
teacher  trainers.  We  have  been  planned  only  1370  participating  teacher  trainers.  Obviously  we  succeed  to 
include  far  more  people. 


Planned 


Realized 


Graph  2.  Number  of  teacher  trainers  attending  training  coursers 

4.3  Number  of  Participating  Schools  and  Counselling 

There  were  1,780  or  99%  of  all  schools  in  Slovenia  included  in  the  project  from  March  2009  to  31  December 
2012.  In  average  there  were  8 visits  of  our  consultants  and  6 on-line  counselling  per  school.  All  together  we 
performed  14,500  live  counselling  and  8,501  on-line  counselling.  Planned  number  of  teachers,  who  would  be 
included  in  counselling,  was  7000.  Actual  or  realized  number  is  36,495. 
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5.  CONCLUSION 

We  set  out  the  path  of  informatization  of  edueational  institutions  in  Slovenia.  Consequently  the  proeess  of 
ehanging  the  traditional  way  of  teaehing  and  ehange  of  teaehing  and  learning  paradigm  itself  has  begun.  This 
is  eertainly  the  way  that  refleets  the  needs  of  modem  soeiety  and  the  students  in  it. 

The  e-Edueation  projeet  is  allowing  eaeh  edueational  institution  to  deeide  and  inelude  itself  into  the 
projeet  aeeording  to  their  needs  and  vision.  The  biggest  sueeesses  of  the  projeet  are:  elear-eut  path  to 
e-eompetenee  of  professional  staff,  pedagogieal  and  didaetie  oriented  seminars  and  eounselling,  possibility  of 
online  independent  assessment  of  e-eompeteneies,  developed  modem  e-leaming  materials,  web  portal  of 
Slovene  Edueational  Network  SIO  and  on-going  evaluation.  It  is  important  to  note  that  sehools’  demands  for 
sueh  teaeher  training  exeeeded  our  plans  and  availability.  It  is  espeeially  eneouraging  that  the  Ministry  of 
Edueation  has  already  prepared  a new  projeet  aimed  at  1 : 1 pedagogy  and  the  development  of  third  generation 
e-textbook. 

The  next  step  would  be  monitoring  effeets  in  sehools.  Only  the  observations  done  in  elassrooms  eould 
prove  the  effeets  of  projeet  and  its  impaet  on  the  way  of  teaehing  and  learning. 
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ABSTRACT 

The  paper  will  discuss  potentials  of  digital  media  to  support  student  engagement  and  student  production  in  Danish  upper 
secondary  education  with  a specific  focus  on  group  work  and  collaboration. 

With  the  latest  school  reform,  upper  secondary  education  in  Denmark  has  experienced  an  increased  focus  on 
problem-based  and  self-governed  work  of  students.  The  paper  is  based  on  a large  research  project  involving  800  students 
and  around  100  teachers  from  15  schools.  The  empirical  study  is  based  on  observations,  interviews  with  teachers  and 
students  from  the  involved  schools  and  on  a survey  with  answers  from  500  students  and  65  teachers.  Within  the  period 
from  2010  to  201 1,  the  involved  schools  have  experimented  with  new  teaching  methods  and  integration  of  different  kinds 
of  digital  media.  Based  on  these  empirical  studies,  the  paper  will  discuss  how  students  can  utilize  digital  media  to  support 
group  work  and  collaboration,  including  joint  production  of  digital  media.  First,  the  paper  briefly  will  present  a system 
theoretical  inspired  approach  to  the  key  concept  of  communication,  teaching  and  knowledge  construction.  Second  the 
empirical  method  and  design  from  the  qualitative  and  quantitative  studies,  based  on  the  system  theoretical  approach  will 
be  described.  Third,  the  findings  from  the  qualitative,  and  quantitative  studies  will  be  presented. 

The  paper  concludes  that  the  study  demonstrates  changes  in  the  way  group  work  is  organised  by  the  students  using  digital 
media,  and  a tendency  to  develop  student  engagement.  Further  more  the  study  shows  evidence  of  increased  transparency 
between  students  and  teachers,  developments  within  student  roles,  a focus  on  new  forms  of  assignments  and  potentially 
exams,  and  increased  openness  and  sharing  among  students.  Hence  innovation  trends  within  digital  media  usage  and 
developments  of  teaching  methods  in  Danish  secondary  education. 


KEYWORDS 

Teaching,  learning,  digital  media,  rethinking  teacher/student  roles  and  teaching  approaches 


1.  INTRODUCTION 

In  Denmark  today,  upper  secondary  schools  are  facing  great  changes  and  challenges.  The  point  of  departure 
is  based  on  an  empirical  study,  which  is  the  first  in  a series  of  four  national  studies  with  a focus  on  rethinking 
teaching  methods  and  student  activities  with  use  of  a range  of  digital  media.  The  current  study  ran  for  two 
years  in  the  period  from  2010-2011,  and  the  school  projects  covered  two  or  three  semesters  depending  on  the 
schools.  The  objective  of  the  studies  is  to  examine  opportunities  and  barriers  in  use  of  digital  media  in 
education,  and  to  contribute  to  a continuing  educational  development  in  upper  secondary  schools. 

The  paper  is  based  on  the  first  empirical  study  and  has  a focus  on  the  research  question:  What  are  the 
opportunities  and  barriers  in  digital  media  in  relation  to  students’  group  work  and  student  engagement?  The 
empirical  analysis’  key  concept  is  communication  and  the  empirical  study  will  examine  technologies  for 
communication,  collaboration,  sharing  and  production  and  discuss  opportunities  and  barriers  as  fare  it  is 
possible  in  this  first  study.  The  involved  schools  have  used  a wide  range  of  technologies  and  have  chosen 
different  approaches  to  their  individual  projects.  The  consequence  is  that  the  study  shows  a variety  of 
teaching  methods  and  technology  use. 
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2.  THE  THEORETICAL  ERAMEWORK 

The  theoretical  foundation  has  been  constructed  on  the  basis  of  Niklas  Luhmann’s  systems  theory, 
particularly  his  concepts  of  social  systems  and  psychic  systems.  This  system  thinking  corresponds  the  way  in 
which  we  observe  the  empirical  study.  The  coupling  of  system  definition  and  system  characteristics  (systems 
as  operative  closed,  self-creating,  self-referential  and  autonomous)  with  the  concept  of  communication  leads 
to  a thinking,  where  it  is  impossible  to  work  with  the  idea  that  thoughts  can  leave  persons  as  thoughts  and  the 
idea  that  knowledge  can  be  transferred  (Luhmann,  1995).  Thus  thoughts,  emotions,  intuitions  and  other 
consciousness  activities  are  observed  in  the  empirical  study  as  communicative  representations,  which  have 
some  decisive  consequences  for,  among  other  things,  a discussion  of  teaching  and  learning. 

2.1  Communication,  Teaching  and  Learning 

The  concept  of  communication  must  be  understood  in  the  system-theoretical  frame  in  which  systems  are 
operative  closed,  self-referential  and  autonomous.  Social  systems  operate  in,  constitute,  reproduce  and 
maintain  themselves  through  communication.  For  instance,  net-mediated  discussion  forum  will  maintain 
itself,  if  there  are  persons  who  constantly  take  part  by  writing  a contribution,  if  not  the  social  system  (the 
forum)  will  slowly  “die”.  Thus,  communication  is  the  minor  element  in  social  systems,  and  this  minor 
element  is  defined  as  the  synthesis  of  three  selections;  the  selection  of  information,  the  selection  of  utterance, 
(made  by  the  “utterer”)  and  the  selection  of  understanding  (made  by  the  “addressee”)  (Luhmann,  1992).  It 
takes  so  to  speak  two  to  tango  - the  point  of  departure  of  the  empirical  study.  Observation  that  involves 
distinction  and  indication  is  another  central  concept.  (Luhmann,  1998,  Spencer-Brown,  1969,  p.  3). 
Multitasking  e.g.  will  not  be  an  option  in  this  theoretical  lens.  All  observations  are  system  internal 
operations,  cf.  the  system  characteristics  above.  Thus,  Luhmann  calls  his  form  of  constructivism  for 
operative  constructivism.  The  communicator,  the  utterer,  undertakes  the  selection  of  information  and  the 
selection  of  utterance,  and  the  addressee  undertakes  the  selection  of  understanding  (Luhmann,  1995,  p.l41). 
The  utterer  and  the  addressee  is  called  psychic  systems,  and  these  operate  in  and  maintain  themselves  via 
consciousness  activities  (thoughts,  emotions,  intuitions  etc.)  which  are  activities  we  do  not  have  direct  access 
to,  cf.  the  point  that  thoughts  etc.  are  communicative  representations.  We  have  to  make  our  own 
construction/interpretation  of  the  uttered  information  by  the  utterer.  This  is  presumably  a truism  for  most 
educators,  however  the  educational  system  often  acts  like  the  transfer  model  of  knowledge  is  still  working. 
Psychic  systems  are  as  well  as  social  systems  based  on  meaning,  implying  that  they  choose  to  actualise 
something  and  let  other  things  rest.  Based  on  the  system  characteristics  already  mentioned,  the  result  is  that 
in  principle  the  individual  system’s  unique  selection  decides  what  the  system  chooses  to  actualise.  That 
includes  us,  as  researchers,  when  we  observe  e.g.  students’  communication  every  observation  is  dependent 
on  the  observer.  For  instance  the  assumption  that  when  a teacher  has  presented  a lesson,  all  the  students  have 
understood  the  same  is  risky  business  in  the  presented  theoretical  frame,  since  the  individual  student  often 
has  no  possibility  of  controlling  comprehension,  if  all  communication  is  one-way  communication  from 
teacher  to  student.  Therefore  it  is  more  likely,  with  the  systems  theoretical  approach,  that  each  student  has 
chosen  his/her  own  understanding,  which  is  not  necessarily  the  same  understanding  as  the  teacher-  and 
his/hers  intention  concerning  the  students’  knowledge  constructions.  In  short,  it’s  all  about  communication 
and  the  premises  for  communication,  when  the  intention  is  to  facilitate  knowledge  construction.  That  is  why 
we  have  chosen  to  focus  on  students’  use  of  digital  media  and  related  activities  concerning  group  work 
settings. 

The  concept  of  teaching  is  regarded  as  communication  that  intends  a change  of  person,  in  the  sense  that 
the  knowledge  construction  may  be  possible  - not  that  it  will  happen  per  se  (Luhmann,  1995).  The  teaching 
environment  can  be  considered  the  perturbing  activity  with  the  intention  of  activating  knowledge 
construction.  The  characteristics  of  the  concept  of  teaching  has  to  be  add  a very  important  detail,  namely  that 
teaching  is  a concept  not  only  concerning  a traditional  teacher-student  relation.  Students  collaboration  with 
student  e.g.  in  groups  we  will  call  teaching  as  well  as  feedback  processes,  guidance,  direction  and 
supervision.  This  is  the  point  of  departure  when  it  concerns  the  selection  of  the  methodological  design  of  the 
empirical  study  and  the  subsequent  analysis. 
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In  this  concept  learning  is  defined  as  the  construction  of  individual  mental  constructions,  while  teaching 
is  understood  as  a special  form  of  communication  intended  to  bring  about  change.  In  brief,  the  result  of 
learning  is  knowledge  construction.  According  to  the  system’s  characteristics  teaching  and  learning  are  two 
different  modes  attributed  to  respectively  social  systems  and  psychic  systems  and  important,  teaching  and 
learning  are  considered  mutually  dependent  (Mathiasen,  2008). 


3.  THE  EMPIRICAL  DESIGN 

The  study  includes  a variety  of  different  teaching  methods.  In  this  paper  we  focus  on  students  activities  when 
they  work  in  groups,  sometimes  characterised  by  being  problem  based  and  self-governed  by  the  students 
sometimes  based  on  tasks  set  up  by  the  teacher.  The  different  kind  of  group  work  settings  gives  different 
premises  for  learning  activities.  Hence  you  can  approach  the  concept  of  group  work  as  a progression  from 
“task  set  up  by  teacher”  to  “self-governed  problem  based  project  work”.  This  study  will  not  focus  on  the 
different  types  of  group  work,  but  centre  around  communication  facilitated  by  a variety  of  digital  media  used 
in  the  group  work,  seen  as  a superordinate  term.  Further  more  - in  consequence  of  teaching  methods 
involving  students  and  the  expectation  of  students’  self-governed  activities  - the  teacher  roles  changes  and 
generation  of  corresponding  teacher  roles  is  observed,  e.g.  moderator,  discussion  partner,  adviser,  supervisor, 
mentor,  counsellor  etc.  (Mathiasen,  2005).  The  empirical  study  was  carried  out  on  the  basis  of  consecutive 
research  design  inspired  by  systems  theory.  Data  collection  methods:  Observations,  group  interviews, 
questionnaires  and  questions  asked  in  plenary  sessions. 

3.1  The  Qualitative  Part  of  the  Study 

This  part  includes  observations  of  the  communication  face-to  face  and  the  web-mediated  communication  in 
different  web-based  forums  used  by  the  students  during  the  group  work.  Furthermore,  the  empirical  study 
includes  two  45  minutes  group  interviews  at  each  school,  with  4-6  students  and  4-6  teachers  respectively. 
The  students  were  selected  by  teachers  from  the  school  project  to  participate  in  the  group  interview.  The 
criteria  was  to  pick  out  students  that  covered  a certain  broadness  of  approaches  to  “being  an  upper  secondary 
school  studenf  ’ and  their  use  of  IT  in  relation  to  the  project.  The  researchers’  school  visits  were  organised  by 
the  same  model  for  all  school  projects.  A meeting  with  teachers  in  the  specific  project  and  some 
representatives  from  the  management  group,  then  observation  in  class,  observation  of  communication,  use  of 
IT  in  different  teaching  settings,  followed  by  student  and  teacher  group  interviews  respectively,  based  on  the 
presented  research  question  and  formed  as  a semi-structured  questionnaire  (Patton,  1990).  Observation  notes 
and  transcribed  interviews  have  - along  with  written  comments  from  the  questionnaires  - been  the  qualitative 
foundation  for  the  thematic  analyses. 

3.2  The  Quantitative  Part  of  the  Study 

Our  study’s  survey  design  consists  of  a questionnaire  distributed  to  the  teachers  involved  in  the  specific 
school  project  and  a more  or  less  similar  questionnaire  for  the  students  involved.  The  two  questionnaires  are 
developed  on  the  basis  of  common  guidelines  and  standards  for  survey  design  and  construction  of  questions 
(e.g.  de  Vans  2002,  Bradburn,  Sudmand  and  Wansink  2004,  de  Leeuw,  Hox  and  Dillman  2008,  Tourangeau, 
Rips  and  Rasinski  2000,  Schuman  and  Presser  1996  and  Foddy  1994).  At  first,  these  two  questionnaires 
identify  which  IT  tools  have  been  used  - and  to  what  extent  - in  teaching  related  activities.  Hereafter,  the 
main  purpose  of  the  survey  is  to  describe  the  potential  for  using  the  applied  IT  tools  in  relation  to  a variety  of 
teaching  methods,  learning  tasks  etc.  from  student  and  teacher  perspectives. 

To  ensure  reliability  and  validity  the  phrasing  of  the  actual  questions  in  the  questionnaires  were  tested  on 
a test  group  of  students  and  teachers.  The  combined  test  consisted  of  observations  during  the  filling  in  of  the 
questionnaire  combined  with  “thinking-aloud”  test  and  a follow  up  “probing”  test  (cf.  Beatty  & Willis, 
2007).  On  the  basis  of  the  conducted  test  a slightly  revised  version  of  the  two  questionnaires  were  distributed 
via  Internet  (mainly  e-mail)  to  the  classes  involved  in  the  project.  All  the  involved  schools  were  asked  to 
work  out  contact  lists  with  e-mail  addresses  on  the  involved  students  and  teachers. 


289 


ISBN:  978-972-8939-88-5  © 2013  lADIS 


3.3  The  Applicability  of  the  Quantitative  Data 

A shade  under  half  of  the  informants  completed  the  questionnaire.  Due  to  the  quite  varying  - and  far  from 
optimal  - response  rates  it  is  obvious  that  we  cannot  generalize  broadly  on  basis  of  these  quantitative  data.  In 
addition  to  this,  premises,  goals  and  settings  for  the  participating  school  projects  are  quite  differentiated  and 
thereby  difficult  to  compare.  Yet  the  quantitative  data  can  provide  us  with  some  indications.  Keeping  the 
above  mentioned  points  and  reservations  in  mind  it  is  our  judgment  that  the  survey,  however,  gives  a useful 
description  of  the  examined  components  for  pointing  out  interesting  tendencies  to  follow  up  upon  in  the 
upcoming  second,  third  and  fourth  round  of  the  research  project. 


4.  NEW  TECHNOLOGIES  EOR  GROUP  WORK-  EMPIRICAL  EINDINGS 

The  study  has  examined  teachers’  and  students’  experiences  with  a range  of  digital  media  in  relation  to 
different  teaching  methods.  The  study  shows  a clear  tendency  that  both  teachers  and  students  give  certain 
types  of  digital  media  a very  positive  evaluation  in  relation  to  group  work.  In  the  study  the  theme 
collaboration  within  group  work  were  highlighted  by  students  and  teachers  as  methods  that  are  especially 
supported  by  digital  media.  This  is  both  supported  by  the  qualitative  and  the  quantitative  studies. 

4.1  Technologies  and  Potential  - Student  Activities 

In  the  quantitative  study  teachers  were  asked  their  opinion  on  the  overall  potentials  of  technologies  and 
digital  media  in  relation  to  different  teaching  methods.  Teachers  found  that  technologies  have  a large 
potential  to  support  the  activities  of  students,  and  that  technologies  were  able  to  increase  student  activity. 


On  the  basis  of  the  IT  project,  please  rate  the  degree  to  which  IT 
tools  contributed  to  increased  student  activity  related  to  teaching. 


Figure  1.  Teacher  ratings  of  IT  tools  in  support  of  increased  student  activity. 

The  qualitative  interviews  confirm  this  opinion  from  the  survey.  Concerning  the  general  conclusion  on 
the  potentials  of  digital  media  to  support  group  work  of  students,  two  teachers  from  different  schools  made 
the  following  statements:  “It  is  evident  that  the  technologies  support  collaborative  working  methods”  and 
“Work  similar  to  project  work  is  promoted  enormously”.  In  the  quantitative  survey,  teachers  were  also  asked 
to  evaluate  a number  of  specific  used  technologies  (such  as  Google  Docs,  TypeWith.me,  Fronter, 
Blackboard,  interactive  whiteboards)  in  relation  to  different  teaching  methods.  The  teachers  evaluated 
Fronter,  FirstClass,  Google  Docs  and  Skype  as  the  most  useful  technologies  to  support  students’  group  and 
project  work.  Students  evaluated  Dropbox,  Fronter,  Google  Docs,  TypeWith.me,  e-books,  and  Wikipedia  as 
the  most  useful  technologies  to  support  their  group  and  project  work. 

4.2  Evaluation  of  Specific  Technologies 

The  survey  asked  students  and  teachers  to  rate  a wide  range  of  technologies  in  relation  to  different  teaching 
methods.  Students  and  teachers  were  only  asked  to  rate  technologies  that  they  had  experiences  with  in  their 
own  classes.  This  section  will  focus  on  the  technologies  that  were  rated  as  the  most  useful  regarding  group 
work.  These  were  Google  Docs,  Dropbox,  typewith.me,  Facebook,  and  wikies.  Teachers  were  asked  to  rate 
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each  technology  within  each  teaching  method  on  the  following  scale:  Absolutely  necessary  / Large  potential  / 
Some  potential  / Limited  potential  / No  potential.  Similarly,  students  rated  the  technologies  based  on  the 
following  scale:  Always  useful  / Often  useful  / Sometimes  useful  / Rarely  useful  / Not  useful. 

In  the  presentation  of  the  findings,  we  have  contrasted  both  teachers’  and  students’  ratings  in  relation  to 
group  work  with  the  ratings  in  relation  to  individual  work.  Google  Docs  was  used  by  a large  number  of 
schools.  Findings  show  that  both  students  and  teachers  see  Google  Docs  as  a useful  technology  for  both 
group  work  and  individual  work.  However,  it  is  also  evident  that  the  potential  for  group  work  is  larger.  From 
the  interviews  we  have  learned  that  some  of  the  more  sceptical  users  have  experienced  problems  of 
accessibility  and  speed  when  using  and  collaborating  in  Google  Docs. 


How  useful  ate  these  IT  tools  when  you  participate  in  group  work  in  yvhen  slurlents  work  in  groups  - Google  Docs 

school?  - Google  Docs 


Figure  2.  Student  (left)  and  teacher  (right)  ratings  of  Google  Docs,  group  work 

The  evaluation  of  wikis  is  not  clear-cut,  and  the  difference  between  potentials  of  group  work  and 
individual  work  is  small. 


Always  Often  useful  Sometimes  Rarely  Not  useful  Don’t  know 
useful  useful  useful 

How  useful  are  these  IT  tools  when  you  participate  in  group  work  in 
school?  - Wikis  (e.g.  Wikispaces  and  other) 


WIren  students  work  in  groups  - Wikis  (e.g.  Wikispaces  and  other) 


Figure  3. Student  (left)  and  teacher  (right)  ratings  of  Wikis,  group  work 

Dropbox  has  been  widely  used  by  many  students  and  teachers.  The  majority  of  both  students  and  teachers 
see  Dropbox  as  a very  useful  technology  both  for  group  work  and  individual  work. 


How  useful  arc  these  IT  tools  when  you  participate  In  group  work  In  vyi,en  students  work  in  groups  - Dropbox 

school?  - Dropbox 


Figure  4.  Student  (left)  and  teacher  (right)  ratings  of  Dropbox,  group  work 
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TypeWith.me  has  been  used  by  fewer  students  and  teachers  compared  to  the  previous  technologies,  but  a 
clear  difference  is  seen  between  the  evaluations  in  relation  to  group  and  individual  work.  Especially  the 
students  evaluate  TypeWith.me  to  be  very  useful  for  group  work,  whereas  the  potential  for  individual  work  is 
more  mixed.  The  same  trend  is  seen  in  the  teachers’  assessment,  however  not  so  clear. 


How  useful  ate  these  IT  tools  when  you  participate  in  group  work  in  yyi,en  students  work  in  groups  - TypeWith.me 

school?  - Typewith.me 


Figure  5.  Student  (left)  and  teacher  (right)  ratings  of  TypeWith.me,  group  work 

The  final  technology  we  wish  to  present  is  Facebook.  It  is  interesting  that  despite  the  criticism  of 
Facebook  for  being  a distractive  element  in  school,  many  teachers  believe  that  Facebook  can  be  a useful 
technology  - especially  in  relation  to  group  work.  The  numbers  from  the  students  are  more  mixed,  but  a 
majority  also  believed  that  Facebook  is  a useful  technology  for  schoolwork,  and  also  among  students, 
Facebook  is  believed  to  have  a larger  potential  for  group  work  than  for  individual  work.  The  qualitative  study 
confirms  these  findings.  Although  students  and  teachers  are  very  aware  of  the  distractive  element  of 
Facebook  in  class,  it  is  also  evident  that  students  utilise  Facebook  for  schoolwork  both  in  and  outside  school 
hours. 


How  useful  are  these  IT  tools  when  you  participate  in  group  work  in  wi,e„  students  work  in  groups  - Facebook 

school?  - Facebook 


Figure  6.  Student  (left)  and  teacher  (right)  ratings  of  Facebook,  group  work 

From  the  interviews  we  learned  that  both  students  and  teachers  observed  Facebook  as  a two-edged  sword, 
given  that  Facebook  invites  students  to  non-teaching  activities,  when  the  intention  is  to  participate  100%  in 
the  group  work. 

4.3  How  do  Technologies  Change  Group  Work? 

In  an  interview,  a teacher  stated  very  clearly  that  it  was  his  experience  that  the  technologies  helped  change 
group  work  in  a way  that  engaged  the  students  more  than  usually.  The  prior  experience  was  that  group  work 
invites  the  motivated  students  to  do  what  is  intended.  Below  we  will  expand  on  this  by  going  into  depth  with 
the  findings  from  the  qualitative  study  with  a focus  on  group  work. 

4.3.1  New  Forms  of  Collaboration 

Especially  the  use  of  collaborative  tools  (such  as  Google  Docs  and  TypeWith.me)  has  contributed  to 
qualitatively  new  ways  of  “contribution  approaches”  to  group  work.  Several  school  projects  have  shown  an 
extensive  use  of  tools  for  student  communication  and  collaboration.  Technologies  have  been  used  both  for 
collaboration  at  a distance  (all  students  with  own  computer),  and  for  group  work  in  class.  Collaborative  tools 
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such  as  Google  Docs  and  TypeWith.me  have  changed  students’  group  work.  Before  the  introduction  of  these 
tools,  students  would  normally  sit  around  the  same  computer,  and  one  of  the  students  would  control  the 
keyboard.  As  a teacher  pointed  out,  oftentimes  the  consequence  of  this  kind  of  group  work  is  that  several 
students  do  not  participate  actively  in  the  group  work. 

In  many  of  the  school  projects,  use  of  collaborative  tools  has  contributed  to  closer  collaboration  within 
group  work,  because  students  can  work  on  shared  documents  synchronously.  However,  this  development 
does  not  mean  that  division  of  work  no  longer  exists.  Using  the  same  collaborative  tools,  some  students 
divide  their  work  whereas  others  work  closely  together.  It  is  the  general  conclusion  of  the  students  that  using 
the  collaborative  tools  gives  them  a better  insight  into  the  work  of  the  other  group  members.  Further,  the 
students  have  experienced  that  they  inspires  by  reading  the  group  work  texts  produced  by  fellow  students  in 
the  group  work.  The  project  schools  have  experienced  that  the  technology  has  brought  new  elements  to  group 
work.  The  collaborative  tools  have  become  a natural  part  of  the  daily  work  of  many  students.  They  use 
TypeWith.me  and  Google  Docs,  they  chat  and  use  Skype  in  their  group  work  (at  home  and  in  school). 

“For  instance,  when  working  in  groups,  we  divide  the  work  between  us,  and  then  we  use  our  own  PC  and 
share  - in  this  case  - a spread  sheet.  It  was  great,  instead  of  just  one  of  us  at  the  computer,  whereas  the  rest  of 
us  would  sit  around  the  screen  watching”  (student) 

A teacher  stresses  the  same  trend:  “The  most  important  advantage  is  that  the  technologies  have 
contributed  to  more  active  group  work.  Without  the  technologies,  oftentimes  75  % of  the  students  do  not 
contribute  to  the  group  work.”  A general  conclusion  regarding  group  work  in  school  is  that  most  students  are 
activated  in  front  of  their  own  computer,  and  that  the  former  trend,  cf.  both  teachers  and  students,  in  group 
work  that  only  a few  students  did  most  of  the  work,  is  reduced.  Although  students  may  still  divide  the  work 
between  them,  all  students  contribute,  not  least  because  the  technologies  make  it  very  visible  what  each 
student  has  contributed  with. 

4.3.2  Increased  Transparency  between  Students  and  Teachers 

Findings  from  the  school  projects  show  that  collaboration  and  sharing  tools  can  support  transparency  in 
students’  school  work.  The  tools  support  new  forms  of  transparency  in  between  students  and  between 
students  and  teachers.  Students’  activities  and  products  are  made  visible  to  fellow  students  and  to  teachers  to 
a greater  extent  than  previously.  Several  schools  show  examples  of  students  using  each  other’s  productions 
and  writings  for  reflection  and  inspiration.  Not  all  students  use  the  works  of  other  students,  but  several 
students  state  that  they  have  benefited  from  reading  works  and  drafts  of  fellow  students. 

Not  only  final  productions  are  made  visible  to  the  class.  Also,  the  process  itself  is  made  transparent 
through  the  technologies.  Through  the  sharing  of  documents  between  students  and  teachers,  it  is  not  only 
possible  for  students  to  follow  the  work-in-progress,  but  it  is  also  possible  for  the  teachers  to  take  part  in  the 
process.  There  is  so  to  speak  new  and  fruitful  conditions  for  the  communication.  Teachers  can  follow  the 
group  and  individual  work  of  students  as  it  happens  - both  in  school  and  outside  school.  It  is  made  easier  for 
teachers  to  take  an  active  part  in  the  group  work  and  provide  feedback  in  the  process  instead  of  afterwards. 
Both  teachers  and  students  have  experienced  that  feedback  during  the  process  is  more  fruitful  than  feedback 
on  the  final  product.  A teacher  states  “Everyone  can  see  the  work  from  all  the  groups.  They  can  see  each 
others’  work  and  get  inspired  by  it”.  Students  have  the  same  experience,  and  they  emphasise  that  the 
collaborative  tools  provide  opportunities  for  students  to  get  inspiration  from  each  other  - also  between 
groups.  As  a student  states:  “It  was  easier  to  see  the  writings  of  the  others  and  to  get  ideas  from  others,  when 
they  wrote  something  useful”.  Several  students  from  different  schools  state  that  they  have  used  the  writings 
of  other  groups  for  help  on  how  to  solve  new  problems  and  assignments.  They  have  not  found  the  direct 
answers,  but  they  have  seen  new  perspectives  on  solving  problems  and  writing  assignments. 

According  to  several  teachers,  an  important  consequence  of  the  collaborative  tools  and  the  increased 
transparency  in  class  is  that  it  has  become  more  difficult  for  students  to  hide  themselves.  The  positive 
perspective  on  this  is  that  teachers  may  be  more  likely  to  identify  the  students  with  academic  problems.  On 
the  other  hand,  students  who  are  insecure  about  their  productions,  writings  and  assignments  do  not  welcome 
this  kind  of  transparent  group  work.  The  involved  schools  have  different  experiences  with  increased 
transparency.  Some  schools  have  developed  a transparent  culture  where  sharing  and  openness  is  widely 
accepted.  Other  schools  experience  resistance  from  some  students  who  do  not  wish  to  share,  either  because 
they  feel  intimidated  by  letting  other  see  their  work,  or  because  they  want  to  keep  their  supposedly  brilliant 
ideas  and  points  for  themselves,  cf.  interviews  with  both  teachers  and  students. 
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4.3.3  Working  on  New  Forms  of  Products 

Traditionally,  group  work  in  upper  secondary  education  in  Denmark  consists  of  written  papers  and  reports. 
Media  production  and  IT  tools  thus  increase  the  possibility  of  the  diversification  of  producing  presentations 
or  reports.  The  result  of  this  diversification  may  also  be  that  quiet  and  shy  students  will  be  more  active. 

“The  use  of  video  supports  another  kind  of  students.  Another  range  of  students  become  active  and  is 
suddenly  able  to  do  a lot  more,  students  who  are  not  normally  visible.  That's  quite  clear”  (teacher). 

There  is  a reported  valuable  outcome  of  producing  audio  and  video  in  foreign  language  education.  As  one 
teacher  states:  “Submission  of  audio  is  some  kind  of  a revolution.  It's  obvious.  There's  no  discussion  about  it: 
It's  a definite  improvement.”  (teacher).  Submission  of  audio  and  video  trains  and  demonstrates  the  students' 
verbal  skills.  A Spanish  language  teacher  concludes  that  the  students  speak  more  freely  and  fluently  than  in 
class:  "It  has  provided  a verbal  freedom.  They  use  their  manuscripts  as  support  and  security  but  they  express 
themselves  more  freely."  One  school  experimented  with  using  the  students’  knowledge  of  media  production 
in  the  form  of  video,  screencasts,  blogs,  and  websites  in  order  to  stimulate  student  activation  and  group 
oriented  work.  In  two  foreign  language  classes  - English  and  Spanish  - video  production  was  a major  part  of 
the  practical  training  of  spoken  language.  In  Spanish  the  writing  and  staging  of  a play  were  assigned  to 
groups  of  students.  As  the  teacher  intended,  the  students  shifted  focus  from  reading  a text  to  performing  a 
play,  thus  creating  fluency  in  pronunciation.  A majority  of  the  groups  chose  to  stage  their  plays  with  puppets 
and  with  the  students  performing  the  dialogue.  According  to  the  teacher  and  the  students  the  off-camera 
position  made  the  students  speak  freely  and  subsequently  they  moved  from  reading  to  performing. 

An  unequivocal  conclusion  on  the  academic  benefit  of  media  production  has  not  been  reached.  However, 
a vast  majority  of  students  state  in  interviews  that  media  production  e.g.  screencasts,  video,  and  websites 
forces  them  to  work  with  and  understand  the  substance  of  the  curriculum.  Likewise,  the  teachers  regard 
media  production  as  a success  considering  the  activating  nature  of  working  with  media. 

4.3.4  A Change  in  Student  Approach  to  Group  Work 

The  qualitative  study  has  shown  that  the  new  forms  of  group  work  bring  challenges  to  student  qualifications. 
It  is  also  evident  that  some  students  do  not  prefer  these  ways  of  working  but  wanted  to  reduce  the 
complexity,  and  preferred  the  traditional  teacher  and  student  roles  and  settings.  Increased  use  of  group  work 
demands  that  students  possess  i.a.  self-discipline  and  are  able  to  carry  out  self-governed  activities.  However, 
it  is  the  experience  of  several  teachers,  that  successful  group  work  has  a positive  influence  on  students’ 
participation,  engagement,  responsibility  and  motivation. 

Several  teachers  have  experienced  that  students  are  more  likely  to  share  their  work  with  fellow  students. 
In  some  schools  this  has  contributed  to  a change  in  student  culture  towards  more  openness.  A side  effect  of 
the  open  culture  is  that  students  often  put  more  effort  into  their  products,  when  they  know  beforehand  that  the 
products  are  shared  with  the  rest  of  the  students. 


5.  CONCLUSION 

The  study  has  shown  that  utilisation  of  digital  media  can  support  new  forms  of  group  work  and  can 
contribute  to  a higher  level  of  student  activity  within  group  work.  The  quantitative  survey  pointed  towards 
specific  technologies  that  are  rated  very  valuable  for  students’  group  work.  The  paper  has  highlighted  the 
potentials  of  digital  media  to  support  synchronous  collaboration  within  groups  and  to  support  transparency 
between  students  - and  between  students  and  teachers  - with  a collaboration  lens  and  not  a control  lens.  The 
paper  has  also  highlighted  barriers  that  relate  to  transparency;  some  students  do  not  like  to  share  their  work 
with  fellow  students.  Finally,  the  study  has  shown  that  an  extensive  use  of  teaching  methods  based  on 
students’  self-governed  and  collaborative  activities  demands  that  students  possess  qualifications  and 
competences  such  as  self-discipline,  responsibility  and  interpersonal  skills. 

These  conclusions  tell  about  opportunities  and  challenges  for  future  upper  secondary  schools.  The 
increased  focus  on  student  collaboration  and  self-governed  group  work  combined  with  utilisation  of  digital 
media  poses  a challenge  for  teachers'  roles  and  pedagogical  competences  using  digital  media.  The  empirical 
study  is  followed  by  three  projects  of  a similar  nature.  These  projects  are  expected  to  contribute  with  more 
knowledge  concerning  both  students’  and  teachers’  roles  and  functions  and  relations  between  different 
teacher  and  student  roles,  in  new  forms  of  teaching  and  student  activity  utilising  digital  media. 
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The  study  points  towards  a number  of  central  keywords  for  a movement  in  Danish  upper  secondary 
education  with  use  of  technology.  Especially  collaborative  work  has  been  highlighted  as  a focal  area  of 
attention.  It  is  not  only  a matter  of  increased  group  work,  but  also  a change  in  the  form.  Students  are 
collaborating  more  closely,  and  teachers  are  able  to  follow  the  collaborative  process  of  the  students  more 
closely.  At  the  same  time,  several  schools  are  experiencing  higher  degrees  of  transparency,  openness  and 
sharing  among  students.  Hence  it  is  about  an  innovative  development  of  the  educational  culture. 
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ABSTRACT 

Theoretical  education  is  mainly  based  on  university  text-books,  which  usually  include  texts  not  structured  according  to 
any  theory  of  text  comprehension.  Structuring  a text  is  a demanding  process.  Text  should  be  organized  and  structured  in 
order  to  include  descriptions  on  micro  and  macro-level  representation  of  the  knowledge  domain.  Since  this  is  difficult, 
diagnosis  of  text  comprehension  can  lie  heavily  on  the  construction  of  the  appropriate  questions  and  dialogue  structure 
about  a not  structured  text.  This  may  also  affect  students’  performance  on  laboratory  education.  Whereas  traditional 
educational  systems  infer  the  reasons  for  the  student’s  behavior  without  directly  involving  the  student,  current 
educational  systems,  supported  by  interactive  learning  programs,  attempt  to  involve  students  in  the  process  of  diagnosis. 
This  paper  presents  the  adaptation  process  of  a non- structured  text  to  ReTuDiS,  an  open  interactive  learning  system 
which  is  being  experimentally  used  today  as  an  diagnostic,  profiling  and  support  system  for  undergraduate  students. 


KEYWORDS 

Text  adaptation;  interactive  learning  environment;  cognitive  theory;  student  profiling;  computer-assisted  education. 


1.  INTRODUCTION 

ReTuDiS  (Reflective  Tutorial  Dialogue  System)  is  an  open  interactive  learning  system  for  reading 
comprehension  based  on  the  theory  of  text  comprehension  Denhiere  & Baudet  (Denhiere  and  Baudet  1992) 
and  dialogue  theory  (Collins  and  Beranek  1986). 

The  theory  of  Denhiere  & Baudet  on  text  comprehension  (Denhiere  and  Baudet  1992)  argues  that  text 
comprehension  implies  the  understanding  of  fundamental  cognitive  categories.  When  a reader  tries  to 
comprehend  a text  constructs  a representation  structure  in  an  attempt  to  reconstruct  the  world  represented  in 
the  text.  In  this  representation  the  key  role  lies  on  cognitive  categories,  the  person,  the  situation,  events,  acts, 
as  well  as  the  temporal,  causal  relations  and  part-whole  relationships  connecting  these  structures 
representation.  The  term  "person"  is  being  used  for  entities  involved  in  the  structure  representation.  The  term 
"situation"  describes  the  state  of  non-existence  of  a change  over  time.  The  term  "event"  describes  an  action 
that  causes  a change  but  not  from  humans.  The  event  may  be  coincidental  or  caused  by  a non-human  entity, 
e.g.  from  a machine.  The  term  "act"  is  an  action  that  causes  a change  but  which  was  due  to  an  interaction  and 
or  input  of  a human.  The  researchers  argue  that  a reader  who  wishes  to  explain  the  operation  of  a technical 
system  constructs  a representation  of  the  "natural  flow  of  things",  where  each  new  event  may  be  causally 
explained  by  the  circumstances  and  effects  of  the  events  that  have  already  happened. 

In  a survey  conducted  Baudet  & Denhiere  themselves  (Collins  and  Beranek  1986),  the  effect  of  the 
structure  of  a technical  text  on  the  comprehension  of  a technical  school  by  the  students  was  being  examined. 
Three  different  groups  of  students  were  given  two  properly  configured  texts  that  refer  to  the  operation  of  a 
car's  starter.  The  two  texts  are  structured  differently.  The  first  of  these  determines  the  relations  and  states  of 
different  units  together  and  then  defines  a sequence  of  events  linking  these  events  temporally  and  causally 
(microstructure).  The  second  text  provides  a hierarchical  structure  of  subsystems  and  uses  between  these 
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subsystems  links  expressing  the  purpose  of  these  relations  (macrostructure).  The  researchers  find  that  the  text 
which  refers  to  the  temporal  and  causal  connection  of  units  helped  on  the  better  comprehension  of  the  text. 

The  theory  of  Collins  (Collins  and  Beranek  1986)  is  related  to  the  objectives  and  dialogue  strategies  used 
by  instructors  when  students  ask  questions,  in  order  to:  a)  the  teaching  of  a subject  and  b)  the  teaching  of 
scientific  thinking.  The  dialogue  is  used  to  serve  both  of  these  two  purposes  of  teaching.  The  trainers  are  not 
trying  to  teach  concepts,  but  bring  them  within  the  context  of  the  theory  of  a subject.  According  to  Collins, 
questions  that  can  feed  a dialogue  in  order  to  achieve  these  objectives  can  follow  the  following  strategies: 

1 . Selection  of  positive  and  negative  examples. 

2.  Case  (or  Comparative)  Selection. 

3.  Selection  of  alternative  examples. 

4.  Production  hypothetical  cases. 

5.  Generation  of  forecasts  / assumptions  by  the  student. 

6.  Verification  cases. 

7.  Student  entrapment. 

8.  Detection  of  consequences  to  a contradiction. 

9.  Supervision  when  submitting  questions. 


2.  RETUDIS  LEARNING  SYSTEM  DESCRIPTION 

The  learning  environment  ReTuDiS  includes  a diagnostic  and  interactive  section. 

Diagnostic  section:  In  the  educational  environment  of  ReTuDiS  the  student  answers  a series  of  questions 
assessing  general  prior  knowledge  (diagnostic  tests)  in  a specific  module  in  the  subject  of  Informatics, 
choosing  from  the  list  of  answers  given.  The  questions  are  designed  to  explore,  according  to  text 
comprehension  theory,  the  comprehension  of  cognitive  categories:  people,  situations  and  events, 
comprehension  of  causal  relations  between  events  and  situations  and  understanding  the  objectives  and 
sub-objectives  of  the  system  for  which  the  particular  system  was  designed  for  (Tsaganou  and  Grigoriadou 
2008,  Self  1993).  Based  on  the  responses  of  the  student,  ReTuDiS  outputs  the  cognitive  profile  of  the  prior 
knowledge  of  the  student  (Caillies  et  al.  2002,  Bull  and  Nghiem  2002). 

Interactive  part:  Based  on  the  cognitive  profile  of  the  learner,  the  interactive  part  of  the  RETUDIS 
provides  personalized  activities  on  one  or  more  of  the  three  built  versions  of  a text,  alongside  with  questions 
and  possible  answers,  as  well  as  feedback.  The  three  versions  of  text  that  includes  each  module  are  R 
(Relational),  M (Transformative)  and  T (Teleological). 

R (Relational)  text  describes  a document  and  questions  which  focus  on  simple  descriptions  of  the 
part-whole  relations  of  the  system  described  in  the  document,  as  well  as  on  descriptions  of  the  processes, 
event  and  system  status.  (Due  to  its  short,  concise  sentences  and  detailed  text,  the  relational  text  is  usually 
better  for  weaker  students  and  beginners). 

M (Transformative)  text  designates  a document  and  questions  which  are  focused  on  describing  the 
sequence  of  events  and  the  state  to  state  transitions  of  the  system.  (The  transformative  text  assumes  some 
basic  educational  background  but  describes  the  system  events  in  detail,  therefore  is  usually  more  suitable  for 
average  students). 

T (Teleological)  text  is  focused  on  detailed  descriptions  of  the  objectives  and  sub-objectives  for  which  the 
system  has  been  constructed,  or  of  the  events  and  mechanisms  which  take  place  within  a system.  (The 
teleological  text  jumps  to  the  results  and  conclusions  without  a thorough  analysis  of  the  system's  events, 
usually  assuming  a strong  educational  background  and  therefore  is  typically  preferred  by  advanced  students). 

After  reading  the  text  and  answer  the  questions  the  system  diagnosis  / assessment  is  presented  to  the 
student  and  he/she  is  asked  to  engage  in  dialogue  with  the  system  in  order  to  revise  the  student  discrepancies 
or  incorrect  answers.  The  student  is  guided  through  the  construction  of  more  coherent  arguments  in  text 
comprehension. 
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3.  ADAPTATION  OF  EDUCATIONAL  MATERIAL  TO  RETUDIS 

The  following  example,  on  the  subject  of:  "Computer  Network  Technology",  describes  how  to  adapt  standard 
theoretical  literature  texts  and  technical  documents  to  those  required  by  the  system  RETUDIS,  or  other 
similar  educational  systems,  for  the  education  of  informatics  students. 

3.1  Cognitive  Category  Types 

In  a text  that  describes  the  function  of  local  network  computers,  the  following  categories  based  on  the 
cognitive  theory  of  comprehension  Denhiere  & Baudet  (1992)  may  be  displayed: 

Individuals  are  the  units  of  the  network  nodes,  computers,  servers,  allocators,  coaxial  cables,  the  operator 
of  the  site,  etc.  For  some  units  synonyms  are  used,  such  as  channel,  pathway  and  pipeline  crossings. 

States  in  the  local  network  is  the  "idle  state"  of  the  network,  i.e.  the  time  that  no  information  is  being 
transmitted,  and  the  "transmission  mode",  i.e.  the  time  during  which  information  is  being  transmitted. 

Act  refers  to  the  state  when  a node  sends  another  message  to  the  bus  in  order  to  communicate  with 
another  node.  Pressing  the  "send"  button  by  the  operator  to  initiate  the  message  from  a sender  node  to 
another  network  node  recipient  is  a human  act.  (Not  mentioned  in  the  text  - implied  act).  This  operation 
initiates  a sequence  of  events  in  the  sender  node. 

Events  occurring  after  the  act  are  in  chronological  order: 

1 - dividing  of  the  message  information  in  packets 

2- attach  control  information  in  each  packet 

3- attempt  to  capture  the  transmission  medium  (channel)  for  transmission 

4- packet  transmission 

5- new  attempt  to  capture  the  channel  after  a while,  if  the  first  attempt  fails 

6- collision  of  packets  sent  by  two  nodes  etc. 

Random  event,  which  it  can  be  an  unpredictable  power  outage  or  any  other  unforeseen  event. 

Temporal  and  causal  relationships  are  relationships  between  events  in  a sequence  of  events.  For  example, 
if  the  attempted  capture  of  the  channel  is  successful  (event  3),  this  results  in  the  transmission  of  the  packet 
(event  4)  (causal,  and  therefore,  also  a temporal  relationship). 

Part-whole  relations.  To  send  a message,  it  is  being  divided  into  packets  from  the  sender  node  using  a 
specific  model  defined  by  the  node.  Each  packet  belongs  exclusively  to  a message.  Part-whole  relations  are 
those  which  form  a sequence  of  events  and  that  their  integration  is  a new  event,  such  as  sending  the  entire 
message  when  finished  sending  the  packets  that  were  divided. 

Micro-level.  At  micro-level,  the  student  who  reads  the  text  regarding  the  functions  of  FAN  computers,  the 
microstructure  gradually  builds  a representation  of  the  text,  defining  the  units  of  the  network,  in  which 
situations  the  network  operates,  which  events  are  causing  the  amendments  to  the  conditions  described  by  the 
text.  The  student  also  fabricates  temporal  and  causal  relationships  and  the  part-whole  relations  connecting 
these  structures. 

Macro-level.  The  following  extract  of  text  describes  how  the  local  computer  network  operates:  "The 
control  information  attached  to  a package  are  necessary  because  they  allow  the  correct  routing  of  the  packet 
into  the  network,  so  as  to  arrive  at  the  right  destination.  Finding  the  shortest  path  interconnecting  any  two 
network  nodes  is  the  problem  of  routing".  The  purpose  for  which  the  control  information  is  being  attached  to 
a packet  is  the  secure  transmission.  Finding  the  shortest  path  is  designed  to  optimize  the  transmission  speed. 
To  reach  an  objective  individual  events  occur  in  order  to  achieve  a sub-objective.  These  events  cause  changes 
to  the  state  of  the  network  and  each  change  is  to  achieve  a purpose.  To  fabricate  the  macrostructure,  the 
reader  should  be  able  to  reconstruct  the  microstructure  of  situations  (e.g.  why  the  network  changes  from 
transmission  to  idle  mode)  and  sequence  of  events  (for  what  purpose  each  event  occurs)  and  part-whole 
relations  (why  a series  of  individual  events  has  to  be  completed)  to  reach  the  predetermined  purpose  (all  this 
done  with  secure  and  fast  communication  in  mind). 
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3.2  Text  Structure 

The  structure  of  a text  describing  a technical  system,  such  as  the  operation  of  a local  computer  network,  must 
enable  the  reader  to  fabricate  representations  of  microstructure  and  macrostructure  of  the  system 
(Grigoriadou  and  Tsaganou  2007).  Specifically,  the  text  should  provide  the  following  descriptions  of  the 
microstructure: 

a)  description  of  the  units  that  compose  the  system  which  the  text  describes  (R) 

b)  description  of  the  part- whole  relations  connecting  the  units  in  the  system  together  (R) 

c)  description  of  static  situations  where  they  can  find  the  units  of  the  system  (R) 

d)  description  of  events  and  complex  sequences  (consisting  of  sequences  of  events)  and  performed  in 
units  of  the  system  (M) 

e)  description  of  causal  and  temporal  relationships  between  events  and  changes  that  cause  the  static 
states  of  the  system  (M) 

f)  description  of  the  part-whole  relationships  between  events,  which  means  the  hierarchy  between 
them  (M) 

Also,  the  text  should  provide  the  following  description  of  the  macro-structure  of  the  representation  of  the 
subject: 

g)  a description  of  the  system  through  a tree  of  objectives  / sub-objectives  for  each  transition  of  the 
system  from  one  steady  state  to  another  (T) 

3.3  Interactive  Activities  Structure 

The  following  activities  are  structured  in  three  categories  based  on  the  above  theory  of  comprehension  and 
related  technical  document  which  describe  the  system  of  the  local  computer  network  (Tsaganou  & 
Grigoriadou,  2010,  Brookshear,  2005). 

1.  Relational  structure  activities.  These  include  descriptive  questions  of  the:  a)  units  which  compose  the 
system  that  the  text  describes,  b)  Part-whole  relationships  that  connect  the  units  of  the  system,  c)  the 
situations  in  which  the  system  may  be  found.  For  example:  In  the  text  there  is  a description  of  the  units  of  the 
network  (e.g.  hubs,  cables),  all  individual  relationships  (e.g.  broadcast  medium  connecting  two  nodes),  a 
description  of  situations  which  may  a network  be  at  (e.g.  data  transmission  mode,  idle)  and  a description  of 
occurring  events  (e.g.  packetization  data,  attach  control  information,  etc.). 

2.  Transformational  structure  activities.  These  include  descriptive  questions  of:  a)  the  functions  which  are 
carried  out  by  events  in  the  units  of  the  system,  b)  the  causal  and  temporal  relationships  between  events  and 
the  changes  which  they  cause  to  the  state  of  the  system  c)  Part-whole  relations  between  events,  i.e.  the 
hierarchy  between  them.  For  example:  In  the  text  there  is  a description  of  a basic  operation  of  the  network, 
which  is  the  data  transmission  and  description  of  the  series  of  events  related  to  this,  as  well  as  the  states  in 
which  the  network  sequentially  switches  to,  the  part-whole  relationship  between  events  (transmission  ends 
when  a series  of  events  is  completed  in  order  to  make  the  reconstruction  of  the  message  by  the  recipient). 

3.  Teleological  structure  activities.  These  include  questions  which  describe  the  system  through  a tree  of 
objectives  / sub-  objectives  for  each  transition  of  the  system  from  one  state  to  another.  For  example:  In  the 
text  there  is  a description  of  sub-objectives  and  objectives  which  need  to  be  achieved.  The  aim  the  secure 
transmission  and  for  this  reason  control  information  is  being  attached.  Another  aim  is  the  rapid  transmission 
and  that  is  why  packets  are  being  routed.  Another  goal  is  to  serve  the  greatest  possible  number  of  users  with 
the  least  possible  resource  cost,  which  requires  a study  of  factors  affecting  network  performance. 


4.  RETUDIS  LEARNING  ENVIRONMENT:  TYPES  OE  QUESTIONS 

For  each  of  the  categories  described  in  the  previous  paragraph,  the  following  types  of  questions  are  being 
suggested: 

1 . Questions  with  alternative  answers. 

2.  Question-pairs  with  alternative  answers:  The  first  question  of  the  pair  (regarding  a position)  requests 
the  student  to  take  a position  on  an  issue.  The  second  question  of  the  pair  (concerning  justification)  asks  the 
student  to  justify  his/her  position  on  the  issue. 
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3.  Categorizing  entities:  The  student  is  asked  to  identify  the  units  described  in  the  text  which  form 
part-whole  relationships  and  make  the  appropriate  connections  so  as  to  describe  the  relationship  between  the 
units.  An  example  of  a question  regarding  categorizing  entities  which  refer  to  a text  based  on  a relational 
description  is  as  follows:  "The  following  terms  which  are  being  encountered  in  the  text  are  given.  Group 
these  terms  into  groups  with  common  characteristics  and  name  each  group.  The  terms  of  each  group  should 
have  a part- whole  relation  to  each  other". 

4.  Classifying  events  in  a sequence:  The  student  is  asked  to  categorize  events  or  functions  in  a sequence 
as  some  of  them  are  misplaced  causally  or  temporally  in  the  list  given.  An  example  of  a question  requesting 
the  classification  of  events  which  may  be  implemented  to  a text  based  on  a teleological  description  is  as 
follows:  "A  list  is  given  with  the  steps  of  a functions  sequence  required  to  reach  an  objective  shuffled. 
Rewrite  the  list  following  the  correct  events  sequence. 

5.  Completing  events  in  a sequence:  The  student  is  required  to  complete  events  or  situations  that  are 
missing  in  a sequence  which  represents  a function. 


5.  EXAMPLES  OE  ACTIVITIES 

5.1  Example  1 

Activity  on  categorizing  entities  which  are  being  mentioned  in  relational  (R)  text.  Title:  "Recognizing  the 
parts  of  a local  network". 

The  following  terms  are  encountered  in  the  text:  twisted  pair  cables,  connectors,  computer,  coaxial  cables, 
connectors,  fiber  optic  terminations.  Groups  these  terms  in  these  categories:  Nodes,  Transmission,  Adaptive 
Material.  This  grouping  shows  the  relations  part-whole  terms  with  groups. 

5.2  Example  2 

Filling  activity  on  events  sequence  which  are  being  mentioned  in  transformative  (M)  text.  Title:  "LAN 
operation  during  packet  transmission". 

In  a network  channel  which  is  at  an  initial  state,  occurs  a sequence  of  events  linked  causally  and 
temporally,  as  well  as  with  part-whole  relationships  which  are  leading  to  the  change  of  the  state  of  the 
network.  The  following  table  shows  the  sequence  of  events  that  occur  when  sending  a message  from  the 
transmission  node  to  the  receiver  node,  which  is  the  system's  default. 

Fill  in  the  missing  events  (steps  4,  6 and  8)  by  selecting  from  the  list  below: 

1 . Initial  (idle)  state:  The  network  is  idling  before  the  packet  transmission. 

2.  Act:  The  user  of  the  transmission  node  requests  the  delivery  of  a package  to  the  receiving  node  by 
pressing  the  "send"  button. 

3.  Event  1:  The  transmission  node  divides  the  package  into  packets,  based  on  the  privileges  it  has 
according  to  the  network  configuration. 

4.  Event  2:  

5.  Event  3:  The  transmission  node  checks  to  see  whether  the  channel  is  available  and  no  other  node 
transmits  a package,  in  order  to  avoid  collisions. 

6.  Transmission  state: 

7.  Event  5:  Comparison  of  the  delivery  address  of  the  transmitted  package  with  the  address  of  the  first 
node. 

8.  Event  6: 

9.  Event  7:  Comparison  of  the  delivery  address  of  the  transmitted  package  with  the  address  of  the  next 
node. 

10.  Event  8:  The  delivery  address  of  the  package  coincides  with  the  address  of  the  receiver  node  and  the 
package  is  being  copied. 

11.  Einal  state  (idle):  Idle  network,  after  the  transmission  of  the  package. 

Selection  of  answers: 

1 . Eorwarding  of  the  packet  to  the  next  node. 

2.  The  transmission  node  attaches  the  destination  address  in  each  packet. 

3.  The  network  channel  is  free  for  packet  transmission  to  both  directions  of  the  channel. 
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5.3  Example  3 

Activity  of  completing  the  missing  parts  of  a document  which  refers  to  a teleological  (T)  educational  text. 
Title:  "Sending  a message  in  a local  network  - Objectives  and  sub-objectives  tree". 

In  this  example  we  have  the  tree  of  objectives  and  sub-objectives  when  sending  a message  via  the  local 
computer  network  topology  bus.  The  corresponding  events  or  sequences  of  events  in  order  to  realize  these 
objectives  may  also  be  seen. 

Suppose  that  the  local  network  contains  three  nodes  (1,2  and  3)  and  the  message  is  sent  from  node  1 in 
order  to  be  received  by  the  node  2.  The  message  is  broken  into  2 packages.  Fill  the  gaps  in  the  table  1 below 
with  events  listed  in  the  following  list  given,  considering  the  sub-objective  to  which  they  relate  and  the  causal 
links  between  sub-objectives  with  events  and  situations. 


Table  1.  Teleological  text  activity  example 


Objectives  & Sub-Objectives  (Macro-level) 

Actions,  states,  events  (Micro-level) 

Objective:  Send  a message 

Idle  state 

Network  idle 

between  two  network  nodes 

Action 

User  presses  the  send  button 

Sub-objective  1:  Improvement  of 
transmission  speed 

Event  1 

Blank 

Sub-objective  2:  Transmission 

security 

Event  2 

Package  addressing 

Event  3 

Blank 

Transmission 
(active  network) 

mode  Transmission  via  the  free  channel 

Event  4 

Blank 

Sub-objective  3:  Control  of 

network  traffic 

Event  5 

Copy  package  A to  node  1 if  it  has  the 
required  address  attached. 

Or 

Or 

Event  6 

Eorwarding  of  package  A to  node  2 if  it  does 
not  has  the  required  address  attached 

Sub-objective  4:  Return  to  the 
original  state 

Idle  state 

Network  Idle  after  package  transmission 

Selection  of  answers: 

1 . Comparison  of  the  addresses  attached  to  the  packet  A and  the  receiving  node  1 . 

2.  Checking  if  a node  transmits  a packet  via  the  bus  at  the  moment.  If  not,  then  the  packet  is  sent, 
otherwise  postpone. 

3.  Dividing  the  message  in  packets  A and  B. 


6.  CONCLUSION 

In  this  paper  the  procedure  for  the  adaptation  of  an  educational  text  to  an  open  interactive  learning 
environment  and  in  line  with  known  text  reading  comprehension  and  dialog  theories  has  been  presented. 
Although  the  examples  covered  a specific  module  and  questions,  the  method  can  be  used  for  the  adaptation 
of  virtually  any  educational  text,  especially  technical  documents,  to  the  environment  of  RETUDIS  or,  with 
adaptations,  to  other  similar  interactive  learning  environments  as  well. 

Via  the  answers  given  by  the  students  and  their  cognitive  profile  formed  by  the  system,  it  is  possible  to 
offer  personalized  support  which  bases  its  recommendations  on  the  performance  of  the  student.  This 
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approach  goes  beyond  the  common  "one-size-fits-all"  approach  which  dominates  education  today  and  can  be 
nullified  via  advanced  learning  environments. 

The  methodology  described  in  this  study  has  been  used  to  adapt  the  educational  texts  of  the  module 
"Computer  Network  Technology",  which  is  today  taught  in  the  department  of  informatics  and 
telecommunications  of  the  University  of  Athens.  The  module  adapted  to  RETUDIS  has  been  successfully 
used  for  a semester  with  very  positive  results,  allowing  the  research  team  to  assess  the  capabilities  of  the 
system,  troubleshoot  potential  problems  and,  ultimately,  vastly  improve  the  system.  Educational  material 
from  several  different  modules  which  are  taught  in  four  different  academic  institutions  and  in  two  different 
languages  is  now  being  adapted  to  RETUDIS,  as  extensive  further  testing  and  research  is  scheduled  to  take 
place  starting  from  the  following  academic  year. 

It  is  noteworthy  to  mention  that  an  open  interactive  learning  environment  is  that  the  student  participates  in 
the  construction  of  his/her  own  cognitive  model,  which  opens  an  unexplored  area  of  research,  that  of  open 
learner  modeling. 
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ABSTRACT 

This  paper  discusses  three  possible  ways  of  applying  podcasts  in  distance  education:  podcasts  of  recordings  of  virtual 
classes,  podcasts  produced  for  specific  pedagogical  aims,  and  podcasts  produced  by  external  organizations.  Through  a 
survey  we  gained  insight  in  the  (until  now  limited)  experiences  of  our  distant  students  with  podcasts,  and  also  in  their 
preferences  for  future  applications  in  education.  Findings  indicate  that  students  prefer  podcasts  that  can  make  them  study 
as  efficiently  as  possible,  such  as  podcasts  with  explanations  of  hard  topics,  illustrations  of  subject  matter  and  feedback 
on  assignments.  Recorded  virtual  classes  and  podcasts  with  elective  subject  matter  are  less  popular.  It  is  claimed  in 
literature  that  podcasts  in  which  students  can  see  and  hear  their  instructor  can  reduce  feelings  of  isolation  and  increase 
feelings  of  connectedness.  There  are  no  indications  that  our  students  have  a distinct  preference  for  such  podcasts. 


KEYWORDS 

Distance  education,  podcasting,  virtual  class,  pedagogical  strategy. 


1.  INTRODUCTION 

In  recent  years,  the  use  of  podcasts  has  increased  dramatically  in  many  fields,  among  them  education.  The 
most  basic  use  in  higher  education  is  to  capture  live  lectures  in  face-to-face  education,  and  to  upload  them  as 
podcasts.  But  podcasts  can  serve  several  other  educational  functions,  as  one  can  read  in  research  studies  and 
practitioners  reports. 

The  focus  in  this  paper  is  upon  the  potential  use  of  podcasting  in  distance  education.  There  are  claims  that 
podcasting  can  have  considerable  benefits  for  distance  education  students.  In  the  cognitive  domain,  podcasts 
can  be  effective  for  clarifying  and  enhancing  understanding  of  subject  material,  especially  because  it  is 
believed  that  a combination  of  different  communication  media  can  be  beneficial  for  the  learning  process  of 
many  students  (Fernandez  et  al,  2009).  Very  promising,  especially  for  distance  education  students,  is  the 
claim  that  use  of  podcasting  also  can  have  positive  effects  in  the  affective  domain.  It  could  reduce  feelings  of 
isolation  and  make  students  feel  more  connected  to  fellow  students  and  instructors,  and  thereby  increase  the 
students’  motivation  (Lee  & Chan,  2007). 

The  objective  of  this  paper  is  to  identify  current  and  potential  uses  of  educational  podcasts  as  a medium  in 
distance  education.  The  results  could  assist  educators  in  making  decisions  where  and  how  to  best  use 
podcasting  in  their  courses.  This  paper  focuses  specifically  upon  two  questions: 

• What  are  promising  ways  of  applying  podcasts  in  distance  education? 

• What  are  the  experiences  of  distance  education  students  with  the  current  use  of  podcasts,  and  what 
are  their  expectations  of  potential  uses? 

In  literature,  one  can  find  numerous  reports  describing  experiences  with  educational  podcasting,  mainly  in 
the  context  of  face-to-face  education.  Based  upon  those  experiences  we  identify  in  this  paper  a number  of 
promising  ways  to  use  this  technology  in  distance  education.  To  answer  the  second  question,  we  collected 
information  from  our  students  by  distributing  a survey  among  them.  These  students  are  enrolled  in  courses  of 
the  Computer  Science  department  of  the  Open  University  of  The  Netherlands  which  offers  only  distance 
education.  The  survey  asked  the  students  about  their  experiences  with  the  current  use  of  educational  podcasts 
and  about  their  interest  in  potential  applications  of  podcasts. 
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For  a meaningful  discussion  about  using  podcasting  in  education,  one  needs  a way  of  categorizing  such 
podcasts.  Several  categorizations  and  taxonomies  can  be  found  in  literature;  see  Heilesen  (2010)  for  a 
discussion.  As  can  be  expected  in  a just  emerging  practice,  they  tend  to  be  tentative  and  incomplete.  In  this 
paper  we  use  the  categorization  of  Figure  1 . It  closely  resembles  the  one  described  by  Hew  (2009)  which  in 
turn  is  based  upon  the  categorization  of  Rosell-Aguilar  (2007).  We  chose  this  categorization,  because  it  is 
suitable  for  our  practical  objective  of  structuring  the  domain  from  the  point  of  view  of  educators  who 
consider  the  use  of  podcast  in  education. 

This  categorization  divides  podcasts  in  two  main  categories:  podcasts  produced  by  instructors  and 
external  podcasts,  produced  by  other  organizations.  The  former  is  divided  into  two  categories:  lecture 
podcasts  and  podcasts  developed  for  specific  pedagogical  functions,  as  assignment  tips  and  post-assignment 
feedback.  Examples  of  external  podcasts  are  podcasts  produced  by  (other)  universities,  by  broadcasting 
organizations  (for  example,  the  BBC)  and  TED-talks. 


Figure  1.  Categorization  of  podcasts  (Based  upon  Hew  (2009)  and  Rosell-Aguilar  (2007) 

We  notice  that  this  categorization  (contrary  to  the  one  by  Hew  (2009)  does  not  include  student-created 
podcasts.  The  reason  for  not  discussing  them  in  this  paper  is  that  one  can  find  in  literature  hardly  any 
experiences  with  student-created  podcasts. 


2.  APPLICATIONS  OF  PODCASTING 

In  this  section  we  present  several  ways  of  using  podcasts  in  education,  and  we  discuss  their  possible  merits 
for  distance  education.  We  also  present  the  students’  opinions  about  those  applications. 

To  learn  the  opinions  of  our  students,  we  distributed  a survey  among  them.  The  survey  consisted  of 
closed  questions,  with  the  possibility  to  make  remarks  or  comments  in  text  fields.  The  questions  measured  in 
the  first  place  the  respondents’  interest  in  potential  applications  of  podcasting.  The  survey  also  included  some 
more  general  questions  about  podcasting.  If  respondents  had  recent  experiences  with  educational  podcasts, 
we  asked  them  about  these  experiences.  Most  of  them  have  limited  or  no  experience  at  all  with  educational 
podcasts. 

We  announced  the  survey  in  different  ways,  in  order  to  find  respondents  with  different  backgrounds  and 
interests.  It  was  made  available  on  the  website  of  the  department,  and  we  also  announced  it  to  students  of 
several,  divergent  courses.  41  respondents  completed  the  survey.  All  of  them  were  students  enrolled  in 
courses  of  the  department  of  Computer  Science. 

The  subsections  are  arranged  according  to  the  categorization  of  Eigure  1.  Therefore  we  present  and 
discuss  in  succession:  the  use  of  recorded  lectures,  the  use  of  podcasts  with  specific  educational  functions 
and  the  use  of  externally  produced  podcasts. 
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2.1  Recorded  Lectures 

The  most  common  application  of  podcasting  in  face-to-face  education  is  to  capture  lectures  and  make  them 
available  online  for  students.  The  benefits  of  such  podcasts  are  well-documented  and  broadly  recognized 
(Abdous  et  ah,  2012;  Heilesen,  2010;  Jowitt,  2008).  Students  use  these  podcasts  for  the  purpose  of  reviewing 
concepts  and  issues  presented  during  lectures.  There  is  clear  consensus  in  literature  that  podcasting  is 
extremely  effective  as  a revision  tool.  Evans  (2008)  found  for  example  that  students  perceive  podcasting  as  a 
more  effective  revision  tool  than  textbooks,  and  consider  it  to  be  more  efficient  than  their  own  notes.  Most 
popular  among  students  is  to  use  these  podcasts  just  before  the  exams,  as  a kind  of  check  whether  they 
understood  the  subject  matter.  According  to  the  students,  this  use  of  podcasts  has  several  benefits  compared 
to  the  face-to-face  lecture,  among  them  the  ability  to  repeatedly  access  the  same  content  and  the  ability  to 
listen  whenever  and  wherever  they  like  (Jowitt,  2009).  Another  finding  is  that  students  see  such  podcasts  as 
additional  resources  rather  than  as  a substitute  for  lectures  and  written  course  materials  (Abdous  et  ak,  2012) 

Podcasts  in  the  form  of  recorded  lectures  can  also  be  used  in  distance  education.  In  our  distance  education 
university  we  offer  synchronous  virtual  classes  (using  Elluminate)  for  many  courses.  These  classes  have  a 
structure  similar  to  face-to-face  classes.  The  instructor  gives  mini  lectures,  introduces  assignments  and  solves 
them  in  interaction  with  a group  of  students.  Usually  a group  consists  of  about  10  or  15  students.  These 
sessions  are  recorded  and  are  available  for  all  students  who  are  enrolled  in  the  course.  Students  who  attended 
the  virtual  class  can  (re)view  them,  but  also  students  who  did  not  attend  the  class. 

We  asked  our  students  about  their  experiences  with  these  recorded  virtual  classes.  Table  1 gives  the 
results.  The  number  of  respondents  is  28,  because  not  all  respondents  had  experiences  with  courses  using 
virtual  classes. 


Table  1.  Use  of  recorded  virtual  classes  (N  = 28) 


Did  you  watch  recordings  of  virtual  classes? 

Seldom  or  never  (<  10%) 

25  % 

Sometimes  (about  25  %) 

25  % 

Regularly  (about  50  %) 

11  % 

Almost  always  (75  % or  more) 

39% 

We  asked  the  students  about  their  interest  for  potential  uses  of  podcasts  in  our  courses,  whether  they  had 
experience  with  these  uses  or  not.  Table  2 gives  the  results.  Entries  7 and  11  are  relevant  for  the  purpose  of 
this  section;  they  present  the  students’  interest  for  recorded  virtual  and  recorded  face-to-face  classes. 

Table  2.  Interest  for  ways  of  using  podcasts  in  courses  in  order  of  preference  (10  pt  scale) 


# 

Topics 

Mean 

1 

additional  explanations  of  hard  topics 

8.2 

2 

illustrations  and  examples  of  subject  matter 

7.4 

3 

how-to-do  instructions  (as  using  tools,  giving 
presentation,  and  so  on) 

7.4 

4 

guest  speakers  (for  example  famous  experts) 

7.3 

5 

feedback  on  assignments 

7.2 

6 

survey  of  subject  matter 

6.9 

7 

recorded  virtual  classes 

6.5 

8 

motivations  why  topics  are  relevant 

6.5 

9 

discussions  between  experts 

6.4 

10 

elective  subject  matter 

6.2 

11 

recorded  face-to-face  classes 

5.9 
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2.2  Specific  Pedagogical  Functions 

In  the  previous  section,  we  discussed  the  reuse  of  recordings  of  virtual  classes.  Such  recordings  can  be  seen 
as  a documentation  of  classes  and  as  an  extension  of  the  practice  of  providing  electronic  versions  of  slides. 
They  are  a kind  of  side  effect  of  giving  a class.  But  instructors  can  also  produce  podcasts  deliberately  as  a 
pedagogical  tool,  with  a focus  upon  specific  pedagogical  functions  and  specific  educational  contexts. 

We  discuss  some  possible  applications  of  this  kind  of  podcasting  in  distance  education.  They  are  largely 
based  upon  experiences  in  face-to-face  education. 

• Podcasts  can  be  used  for  structuring  the  study,  to  prepare  students  for  the  core  learning  activities.  For 
example,  Fernandez  et  al.  (2009)  describe  how  short  podcasts  can  give  students  a lead-in  to  and  an 
outline  of  each  chapter  of  a textbook,  a motivation  why  specific  topics  are  included,  and  hints  for 
study.  Such  podcasts  are  relevant  for  distance  education  students,  because  they  could  support  them  in 
their  efforts  to  efficiently  manage  their  time. 

• Podcasts  can  prepare  students  for  examinations.  For  example,  the  instructor  can  look  back  on 
learning  objectives,  discuss  previous  exams  and/or  summarize  the  main  topics.  From  studies  about 
the  use  of  recorded  lectures  in  face-to-face  education  we  know  that  students  view  these  podcasts  in 
the  first  place  to  prepare  themselves  for  the  examinations  (Fernandez  et  ah,  2009;  Heilesen,  2010; 
Kay  2012). 

• Podcasts  can  give  additional  explanations  of  subject  matter  known  to  be  hard  for  students  to 
understand.  Viewing  and  reviewing  difficult  topics  is  also  known  to  be  a popular  way  of  using 
recorded  lectures  (Heilesen,  2010) 

• Podcasts  can  give  feedback  to  students.  For  example  instructors  can  review  learning  outcomes, 
comment  upon  electronic  discussions,  or  give  feedback  to  assignments.  Offering  feedback  by 
podcasts  might  give  students  a more  precise  and  detailed  understanding  of  the  teachers’  comments 
than  offering  feedback  face-to-face  (Heilesen,  2010).  Moreover,  podcasts  can  be  viewed  repeatedly, 
which  is  a substantial  advantage  compared  to  face-to-face  feedback  (Elliot  et  ah,  2009). 

• Podcasts  can  offer  interviews,  for  example  with  experts  in  a specific  field,  writers  of  textbooks,  or 
with  different  stakeholders  in  student  projects,  as  clients  and  users  in  real-world  projects. 

• Podcasts  can  explicitly  be  meant  to  promote  students  to  be  active  while  learning.  Students  tend  to 
view  educational  podcasts  in  a relatively  passive  manner,  which  has  been  denoted  as  receptive 
viewing  (Kay,  2012).  It  is  therefore  a challenge  to  find  ways  to  use  podcasts  to  promote  active 
student  engagement,  especially  in  distance  education  (Abdous  et  ah,  2012).  One  of  the  possibilities  is 
to  adapt  a common  practice  in  face-to-face  classes:  to  alternate  mini-lectures  and  student  activities.  In 
distance  education,  the  mini-lectures  of  the  instructor  can  be  replaced  by  podcasts  covering  the  same 
subject  matter.  To  persuade  students  to  become  active,  each  of  these  podcasts  could  result  in  an 
explicit  invitation  to  solve  an  assignment  or  to  start  discussions  about  the  subject  matter. 
There  are  other  examples  of  the  use  of  podcasts  in  face-to-face  education  to  promote  student  activity 
that  possibly  can  be  adapted  to  distance  education.  One  of  them  is  the  use  of  podcasts  in  step-by-step 
procedures  to  solve  specific  problems,  for  example  in  mathematics  (Kay  & Kletskin,  2012).  Another 
is  the  use  of  podcasts  in  the  ‘inverted  classroom’,  in  which  all  in-class  lecturing  is  replaced  by 
podcasts  and  class  hours  are  used  for  student  activities  (Heilesen,  2010). 

In  the  survey  we  asked  the  students  to  score  their  interest  in  podcasts  with  the  following  functions: 

• survey  of  learning  unit 

• additional  explanation  of  hard  topics 

• illustrations  and  examples 

• feedback  on  assignments 

• motivation  of  importance  of  subject  matter. 

Table  2 gives  the  results  in  entries  1,  2,  5,  6 and  8. 

We  conclude  this  section  with  two  observations.  First,  many  of  the  pedagogical  functions  mentioned  can 
also  be  realized  with  other  media,  for  example  with  texts  only.  But  Fernandez  et  al.  (2009)  report  that 
students  believed  that  a combination  of  different  communication/learning  media  (voice,  text,  pictures,  etc.) 
could  improve  the  results  of  their  learning  processes,  because  they  allow  students  to  learn  in  different  ways. 
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It  has  been  claimed  and  suggested  that  podcasts  can  not  only  have  benefits  in  the  cognitive  domain,  but 
also  in  the  affective  domain  (Kay,  2012).  An  interesting  example  is  using  podcasts  that  are  produced  by  the 
actual  instructor  of  a course.  By  viewing  these  podcasts  students  see  and  hear  their  instructors,  which  could 
add  to  the  feeling  of  proximity  and  presence  of  the  instructors.  For  distance  education  students  this  might  be 
a very  interesting  feature  of  podcasts,  because  it  could  increase  students’  motivation.  We  will  discuss  this 
topic  in  section  3. 

2.3  External  Podcasts 

On  the  web  one  can  find  numerous  potential  educational  resources,  which  can  be  used  for  all  kinds  of 
purposes.  Several  universities  provide  complete  courses  (Massive  Online  Open  Courses  or  MOOCs),  TED 
offers  numerous  interesting  talks  by  well-known  experts,  YouTube  and  other  sites  offer  practical  how-to-do 
podcasts.  There  are  resources  that  can  be  linked  directly  to  specific  subject  matter  and  learning  goals,  other 
resources  can  be  used  to  motivate  and  inspire  students.  Examples  of  the  second  category  in  the  domain  of 
Computer  Science  are  well-known  monologues  of  Steve  Jobs  and  Randy  Pausch. 

The  survey  asks  the  students  to  score  their  interest  to  see  podcasts  with 

• guest  speakers,  for  example  famous  experts 

• relevant  discussions  between  experts. 

Table  2 gives  the  results;  see  entries  4 and  9. 

A tricky  point  in  using  external  podcasts  is  that  they  are  licensed  under  different  terms.  The  fact  that  a 
podcast  is  freely  downloadable  does  not  mean  it  can  be  used  in  any  way.  It  can  be  a complex  matter  to  find 
out  if  and  how  specific  podcasts  can  be  used  for  educational  purposes. 


3.  FEELING  PROXIMITY  AND  PRESENCE 

Lack  of  interaction  is  a key  problem  for  distance  students.  Distance  education  is  often  experienced  as  a lonely 
activity.  As  a result  many  students  are  confronted  with  motivational  problems.  They  feel  isolated  and  less 
member  of  a group,  compared  to  students  in  face-to-face  classes.  They  have  fewer  possibilities  to  interact 
with  fellow  students  and  faculty,  which  can  decrease  their  motivation  and  enthusiasm  (Lee  & Chan,  2007). 

A relevant  question  is  whether  instructor-made  podcasts  can  play  a role  in  alleviating  those  problems  of 
distance  education  students.  Eor  example,  could  the  ability  to  see  and  hear  the  instructor  and  fellow  students 
increase  their  motivation?  There  are  suggestions  and  claims  that  podcasting  indeed  can  be  effective  in 
reducing  feelings  of  isolation  and  in  promoting  a sense  of  belonging  to  a community,  and  therefore  is  able  to 
increase  distance  students’  motivation.  Eemandez  et  al.  (2009)  provided  students  in  an  online  course  with 
podcasts  of  instructors  presenting  surveys  of  relevant  topics.  One  of  their  major  findings  was  that  these 
podcasts  increased  the  feeling  of  proximity  between  students  and  teachers,  because  students  had  the  feeling 
of  a permanent  presence  of  their  teachers.  As  a consequence,  the  podcasts  enhanced  students’  motivation.  In 
a review  study  Hew  (2009)  it  is  suggested  that  the  use  of  podcasts  could  create  a greater  sense  of  rapport  and 
intimacy,  realism  and  motivation.  Lee  & Chan  (2007)  found  that  instructor-made  podcasts  were  effective  in 
reducing  isolation-induced  anxiety  and  in  promoting  a sense  of  inclusivity  and  of  belonging  to  a learning 
community  for  distance  education  students. 

Our  survey  included  questions  about  who  the  students  like  to  see  in  podcasts.  Some  of  the  respondents 
had  some  experience  with  seeing  their  instructor,  because  they  had  some  experience  with  seeing  their 
instructor,  because  they  had  viewed  podcasts  of  virtual  classes.  Table  3 gives  the  results. 

Table  3.  Who  do  you  like  to  see  in  podcasts?  (5-pt  scale,  l=disagree,  5=agree) 


I would  like  to  see  famous  experts  in  podcasts. 

3.6 

It  is  a plus  if  you  can  see  your  instructor  in  podcasts. 

3.0 

I don’t  care  who  speaks,  more  important  is  that 
he/she  has  a convincing  story. 

4.2 
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4.  GENERAL  QUESTIONS 


The  survey  included  some  general  questions  about  the  use  of  podcasts.  First  we  asked  the  students  who 
watched  podcasts  if  they  proved  to  be  useful.  The  response  can  be  found  in  Table  4. 

Table  4.  Usefulness  of  podcasts  (N=22,  5-pt  scale,  l=disagree,  5 =agree) 


The  podcasts  I watched  proved  to  be  useful. 


3.6 


We  asked  the  same  students  if  they  experienced  problems  using  the  podcasts.  Table  5 gives  the  results. 


Table  5.  Experienced  problems  (N  = 14,  5-pt  scale,  l=seldom,  5=often) 


Did  you  experience  problems  with  podcasts? 

Could  not  watch  podcast. 

1.7 

Podcast  lasted  too  long. 

2.3 

Visual  quality  was  bad. 

1.6 

Sound  quality  was  bad. 

1.9 

Podcast  had  no  clear  relation  with  subject  matter. 

1.9 

What  role  should  podcasts  play?  One  of  the  findings  in  face-to-face  education  is  that  students  see 
recorded  lectures  not  as  a replacement  of  face-to-face  lectures,  but  as  an  additional  facility  (Fernandez  et  ak, 
2009;  Heilesen,  2010).  In  the  survey  we  asked  the  students  a similar  question  (Table  6). 


Table  6.  Role  of  podcasts 


What  role  should  podcasts  have? 

Podcasts  should  introduce  no  new  content,  only 
same  content  in  another  way. 

86% 

Podcasts  might  introduce  new  content  that  is  not 
available  in  another  form. 

14% 

Finally,  we  asked  the  students  if  they  felt  the  university  should  increase  the  use  of  podcasts  in  courses 
(Table  7). 


Table  7.  More  podcasts?  (5  pt-scale,  1 = disagree,  5 = agree) 


University  should  increase  using  podcasts  in  courses 


3.2 


5.  DISCUSSION 

According  to  the  preferences  that  are  ranked  in  Table  2,  our  students  prefer  podcasts  that  are  to-the -point,  and 
directly  related  to  the  subject  matter  they  have  to  study.  This  holds  for  4 of  the  5 top  rankings  of  Table  2: 
additional  explanations  of  hard  topics,  illustrations  or  examples  of  subject  matter,  how-to-do-something 
demonstrations  and  feedback  by  the  instructor.  This  preference  is  for  many  distance  educations  students  in 
accordance  with  their  situation,  in  which  family  obligations,  work  and  study  compete  for  attention.  For  such 
students  a natural  tendency  is  to  strive  for  tools  and  methods  that  can  support  them  in  their  endeavor  to  study 
optimally  and  efficiently,  without  wasting  time  to  superfluous  activities.  This  preference  of  our  students  is 
consistent  with  the  findings  of  Fernandez  et  al.  (2009).  Their  study  showed  that  one  of  the  most  important 
ways  that  podcasts  helped  their  (distance  education)  students  was  by  allowing  them  to  efficiently  manage 
their  time.  The  podcasts  gave  them  the  feeling  that  the  concepts  of  the  course  were  easier  to  assimilate. 
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From  the  instructors’  point  of  view,  creating  podcasts  that  are  tailored  to  the  needs  of  distance  education 
students  can  be  difficult  and  time  consuming  (Carvalho  et  ah,  2009)  A cheap  and  easy  alternative  is  to  record 
virtual  classes  (or  face-to-face  classes)  and  make  these  available  as  podcasts  for  all  enrolled  students. 
Students  in  our  department  have  experiences  with  such  podcasts.  According  to  the  survey,  some  of  them 
watch  these  podcasts  frequently,  others  only  occasionally.  From  Table  2,  which  contains  the  preferences  for 
podcasts,  we  can  see  that  their  interest  in  recorded  classes  is  in  line  with  their  experiences.  Such  podcast  are 
appreciated  by  some,  but  by  and  large  they  don’t  rank  high.  Recorded  classroom  sessions  are  not  aimed  at 
emphasizing  time  on  task,  and  Fernandez  et  al.  (2009)  suggest  that  that  could  be  a reason  why  recorded 
classes  do  not  score  high  among  distance  education  students. 

The  benefits  of  recorded  lectures  for  distance  education  students  remain  yet  to  be  proven.  Just  uploading 
any  recorded  lecture  might  not  be  the  best  thing  to  do.  For  example,  Sebastian  Thrun,  one  of  the  pioneers  in 
the  field  of  Massive  Open  Online  Courses  (MOOCs)  sees  recorded  lectures  as  deeply  flawed.  Lectures  are  in 
general  “boring,”  he  says,  and  recorded  lectures  are  even  less  engaging:  “You  get  the  worst  part  without 
getting  the  best  part.”  (Carr,  2012). 

Maybe  the  use  of  such  recordings  can  become  more  attractive  if  instructors  adapt  them  slightly.  One  issue 
is  the  length  of  such  podcasts.  Usually  a virtual  class  session  lasts  for  about  an  hour  or  more,  and  usually 
several  topics  are  covered  within  such  a session.  From  literature,  it  is  well  known  that  students  prefer  short 
podcasts  about  demarcated  topics.  Table  5 shows  that  our  students  scored  the  length  of  the  recorded  virtual 
classes  as  the  largest  ‘problem’  with  podcasts  (although  still  not  as  a big  problem).  It  is  not  possible  to  give 
an  optimal  length,  because  this  presumably  depends  upon  the  subject,  the  presentational  qualities  of  the 
speaker  and  the  attitude  of  the  student  (Hew,  2009).  But  in  general,  podcasts  of  5 or  10  minutes  are  popular. 
Therefore,  it  seems  advisable  to  break  up  long  podcasts  in  smaller,  coherent  segments.  Another,  more 
laborious  adaptation,  is  to  punctuate  the  short  segments  by  on-screen  exercises  and  quizzes.  ‘Peppering’ 
students  with  questions  might  keep  them  active  and  involved  with  the  content,  while  providing  the  kind  of 
reinforcement  that  has  been  shown  to  strengthen  comprehension  and  retention  (Carr,  2012).  It  is  a challenge 
to  devise  types  of  podcasts  in  a distance  education  context  that  stimulate  students  to  be  active. 

Many  students  are  positive  about  using  external  podcasts,  for  example  podcasts  with  famous  guest 
speakers.  Such  podcasts  usually  do  not  match  seamlessly  the  subject  matter  of  a corresponding  course.  To 
give  optimal  support  to  students  that  strive  for  maximal  study  efficiency  it  is  advisable  to  thoughtfully 
integrate  such  external  podcasts  into  the  (other)  course  materials.  This  can  be  done  by  offering  clear  and 
exact  information  about  what  they  can  expect  from  each  podcast:  its  length,  the  topic  it  does  support,  its  exact 
relationship  with  the  topic,  and  its  relevance.  In  this  way,  every  student  can  make  a well  informed  decision 
whether  or  not  to  use  it. 

It  has  been  claimed  that  podcasts  can  add  to  the  feeling  of  proximity  and  presence  of  the  instructors, 
which  could  increase  student  motivation.  Up  to  now  our  students  have  not  indicated  they  expect  much  of  the 
possibility  to  be  able  to  see  their  instructors  in  podcasts.  This  is  consistent  with  the  finding  that  recorded 
classroom  sessions  have  not  high  rankings  in  Table  2,  with  the  preferred  uses  of  podcasts.  But  it  has  to  be 
stressed  that  these  opinions  are  based  mainly  upon  expectations,  not  upon  experiences.  Practice,  of  course, 
may  be  different  from  the  students’  expectations. 

Currently,  our  students  see  podcasts  in  addition  to  written  course  materials,  not  as  a substitute  of  these. 
This  is  in  line  with  one  of  the  main  findings  of  Fernandez  et  al.  (2009),  who  found  that  podcasts  are 
perceived  as  a complement  to  traditional  resources  of  a course,  not  as  a substitute  for  them. 

Overall,  the  respondents  have  rather  mixed  opinions  about  the  use  of  podcasts  in  education.  On  the 
question  whether  the  university  should  increase  using  podcasts,  the  mean  score  is  rather  neutral  (Table  7). 


6.  CONCLUSIONS 

We  distinguished  three  categories  of  applications  of  podcasts  in  (distance)  education: 

• podcasts  of  recordings  of  virtual  classes  that  are  made  available  for  all  students  who  might  have  an 
interest  in  them 

• podcasts  that  are  produced  with  a specific  pedagogical  function  in  mind,  for  example  instructors 
summarizing  or  reviewing  a topic,  giving  feedback  or  explaining  hard  topics 

• podcasts  produced  by  external  organizations,  for  example  guest  speakers  of  famous  universities. 
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We  identified  some  topics  students  like  to  view  as  educational  podcasts.  The  respondents  preferred 
podcasts  that  are  to-the-point,  directly  related  to  the  subject  matter  they  have  to  study,  and  that  can  make  the 
difference  to  study  efficiently.  On  top  of  the  list  of  preferences  are  podcasts  that  are  produced  with  a specific 
pedagogical  function  in  mind.  The  first  three  are:  podcasts  with  additional  explanations  of  hard  topics, 
illustrations  or  examples  of  subject  matter  and  how-to-do-something  demonstrations. 

An  inherent  characteristic  of  podcasts  that  are  tailored  to  the  needs  of  students  is  that  producing  them 
usually  is  time-consuming.  Instructors  have  limited  time,  and  they  must  be  effective  and  efficient.  Therefore, 
they  might  consider  using  recordings  of  virtual  classes.  A cheaper  alternative  is  to  use  recordings  of  virtual 
classes.  But  in  the  ranking  of  preferred  podcasts  they  do  not  score  high,  perhaps  because  they  are  not  enough 
focused  upon  time  on  task.  Nonetheless,  a considerable  number  of  students  appreciate  such  podcasts  highly, 
so  they  should  certainly  be  offered  if  they  are  available.  Besides,  it  is  possible  to  increase  their  attractiveness, 
for  example  by  dividing  recordings  in  small  pieces. 

Students  have  an  interest  in  external  podcasts.  For  example,  many  of  them  like  to  see  famous  experts.  It  is 
advisable  when  using  such  podcasts  to  give  enough  information  about  them,  for  example  about  their  exact 
relation  with  the  subject  matter  and  about  the  length  of  them.  Provided  with  such  information,  students  can 
make  a well  informed  decision  whether  or  not  to  use  them. 

We  did  not  find  much  support  for  the  claim  that  podcasts  can  support  distance  students  by  adding  to  the 
feeling  of  proximity  and  presence  of  the  instructors.  Our  students  have  not  indicated  they  expect  much  of  the 
possibility  to  be  able  to  see  their  instructors  in  podcasts. 

It  has  to  be  stressed  that  the  opinions  of  the  students  are  largely  based  upon  expectations,  not  upon 
experiences.  Practice,  of  course,  may  be  different  from  the  students’  expectations.  In  future  research,  we 
would  like  to  find  out  if  the  actual  use  of  different  kinds  of  podcasts  leads  to  the  same  results. 
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ABSTRACT 

In  the  information  age,  one  of  the  most  influential  institutions  is  education.  The  recent  emergence  of  MOOCS  is  a sample 
of  the  new  expectations  that  are  offered  to  university  students.  Basing  decisions  on  data  and  evidence  seems  obvious,  and 
indeed,  research  indicates  that  data-driven  decision-making  improves  organizational  productivity.  The  most  dramatic 
factor  shaping  the  future  of  higher  education  is  Big  Data  and  analytics.  Big  Data  emphasizes  that  the  data  itself  is  a path 
to  value  generation  in  organizations  and  it  is,  also,  a critical  value  for  higher  education  institutions.  The  emerging  practice 
of  academic  analytics  is  likely  to  become  a new  useful  tool  for  a new  era.  Analytics  and  big  data  have  a significant  role  to 
play  in  the  future  of  higher  education.  This  paper  attempts  an  analytical  practice  about  the  use  of  e-learning  technological 
tools  to  generate  relevant  information,  for  the  teacher  and  the  students  who  try  to  optimize  their  learning  process.This 
combination  of  data-processing  and  analytical  learning  is  an  aid  to  improve  significantly  higher  education  and  mark  the 
path  to  follow  in  the  new  educational  era. 
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1.  INTRODUCTION 

Nowadays,  there  is  a huge  amount  of  data  from  the  students  that  are  accessing  Learning  Management  System 
(LMS).  The  increase  of  students  in  the  new  educational  systems  in  distance  learning  is  causing  a new  trend  in 
the  world  of  education.  The  recent  emergence  of  MOOCS  {Massively  Open  Online  Courses)  is  a sample  of 
the  new  expectations  that  are  offered  to  university  students. 

This  trend  leads  to  a role  change  in  the  behavior  of  the  different  educational  agents,  where  both,  teachers 
and  pupils,  must  conform  to  the  new  methods  and  change  their  traditional  methods  of  teaching[l  1].  In  this 
phenomenon  nor  is  it  beyond  the  role  of  academic  institutions  that  are  in  the  obligation  to  modify  its 
structures  and  their  information  systems  to  meet  the  students  needs  in  order  to  have  access  to  their  academic 
offer. 

Is  there  anything  we  can  do  with  the  vast  amount  of  data  provided  by  students  to  improve  the  educational 
system?  Until  recently,  there  were  no  storage  techniques  that  enable  the  analytical  study  of  the  information 
present  in  the  Learning  Management  Systems.  Now  there  are  more  and  more  new  analytical  methods  that 
allow  us  to  deal  with  the  study  of  these  data  and  infer  trends  of  the  use  that  the  students  make  with  respect  to 
the  tools  available  in  platforms.  The  implementation  of  these  analytical  methods  is  possible  through  the  use 
of  powerful  new  technologies  such  as  Data  Mining  or  Big  Data  that  enable  the  processing  of  large  amounts 
of  information  by  searching  discover  new  knowledge  that  is  present  in  the  data. 

It  is  in  this  context  that  this  paper  is  framed,  reflecting,  through  the  analytical  study  of  the  data  current  on 
a LMS  the  use  of  that  students  make  of  the  tools  that  are  available  in  a learning  platform.  This  study  provides 
relevant  information  that  will  allow  teachers  to  optimize  the  use  of  the  tools,  to  deal  with  in  depth  of  the 
learning  strategies  that  are  currently  applied  to  the  university  teaching. 
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2.  BIG  DATA/DATA  MINING:  TECHNOLOGY  FOR  DATA  ANALYSIS 

The  use  of  technological  tools  in  education  in  general,  and  in  all  levels  of  the  individual  learning,  has 
followed  such  a rapid  pace.  If  something  has  marked  the  progress  of  advanced  societies,  over  the  past  few 
decades,  has  been  the  remorseless  development  and  massive  use  of  technological  tools  to  manage  all  kinds  of 
tasks[l].  The  processing  of  the  large  amount  of  existing  data  in  the  field  of  education  has  been  made  possible 
thanks  to  the  development  of  new  Information  and  Communications  Technologies  (ICT).  This  development 
has  led  to  diverse  educational  institutions  carry  out  an  analysis  of  existing  data  from  the  interactions  of  its 
students,  and  draw  conclusions  that  will  improve  the  working  environment. 

To  analyze  this  immense  amount  of  information  is  beginning  to  use,  increasingly,  two  treatments  or 
processes  known  as  Data  Mining  and  Big  Data.  Data  Mining,  also  known  as  KDD  process  {Knowledge 
Discovery  Databases),  is  a process  that  allows  you  to  discover  hidden  information  in  large  volumes  of  data. 
In  the  course  of  the  process  it  works  with  data  subsets,  looking  for  similar  patterns  of  behavior  or  predictive 
models  that  can  be  inferred  from  the  processed  data. 

While  its  use  began  with  economic  purposes,  their  multiple  possibilities  have  allowed  us  to  extend  its  use 
to  the  field  of  education.  The  main  methods  used  and  their  key  applications  are[2]: 

• Prediction'.  Develops  a model  to  infer  some  aspects  of  the  data.  It  is  used  to  emulate  the  behavior  of 
students  in  function  of  their  previous  activities  and  to  predict  the  possible  outcomes. 

• Clustering:  Looking  for  classifying  data  into  groups  with  the  same  characteristics.  Lets  you  know 
common  patterns  for  students  who  are  in  the  same  group. 

• Relationship  Mining:  Discover  relationships  between  variables.  Allows  you  to  discover  associations 
of  activities  that  can  induce  a sequencing  of  the  same.  It  also  highlights  the  most  effective 
pedagogical  strategies  in  the  learning  process. 

• Visualization:  Allows  you  to  discover  trends  in  the  use  of  educational  platforms  that  are  outside  of 
the  average  of  students,  known  as  data  noise. 

However,  with  the  irruption  of  MOOCS  the  online  information  storage  is  growing  in  such  a way  that  the 
processes  for  managing  this  information  begin  to  stay  small,  causing  a serious  problem  not  being  able  to 
exploit  the  data  with  the  necessary  guarantees.  In  order  to  be  able  to  process  such  information  is  necessary  to 
have  new  methods,  being  Big  Data  the  last  to  be  applied  to  the  learning  area.  The  term  Big  Data  generates 
even  confusion  and  is  used  to  associate  concepts  relating  to  large  amounts  of  data,  sociological  analysis,  data 
management  capabilities  and  recently,  also,  in  the  education  area. 

Big  data  can  be  defined  as  the  digital  convergence  of  structured  data  found  inside  databases,  and 
unstructured  data  flowing  from  new  sources  like  social  networks,  mobile  devices,  sensors,  RFID,  smart 
meters  and  financial  systems[6].  Otherwise,  the  Mckinsey  Global  Institute  defines  big  data  as  ‘"datasets 
whose  size  is  beyond  the  ability  of  typical  database  software  tools  to  capture,  store,  manage  and  analyze”[S]. 
This  method  allows,  today,  that  organizations  can  capture  and  analyze  any  data,  regardless  of  what  type,  how 
much,  or  how  fast  it  is  moving,  and  make  more  informed  decisions  based  on  that  information[9]. 

We  recognized  that  there  is  still  got  a lot  to  learn  about  how  to  work  with  big  data,  just  like  everyone  else. 
But  one  thing  we  know  for  sure  is  that  the  traditional  ways  of  working  with  data  will  not  lead  to  success  in 
big  data  analytics [12].  The  variety  of  information  sources,  the  volume  of  information,  latency  of  processing, 
even  the  basic  business  models  are  often  all  different  in  the  big  data  space.  Someone  who  recommends  using 
the  same  old  tools  under  these  new  circumstances  is  someone  who  is  outside  of  the  data  analysis. 


3.  WHAT  IS  MEANT  BY  LEARNING  ANALYTICS? 

It  is  undeniable  that  the  technological  E-learning  tools  have  come  to  play  a decisive  role  in  the  dissemination 
of  the  knowledge.  The  LMS  allows  you  to  capture  explicit  data  through  the  activity  carried  out  by  students 
from  any  device  and  provide  a clear  knowledge  of  what  is  really  happening  in  the  learning  process  in  order  to 
be  able  to  meet  and  enhance  the  care  and  the  needs  of  the  students.  The  importance  of  the  LMS  has  done  that 
many  countries  are  investing  their  effort  in  educational  systems  in  order  to  address  the  challenges  required.  In 
its  review  they  are  using  academic  analytics  working  with  large  data  sets  and  predictive  models  to  practice  an 
institutional  data  mining  to  produce  intelligent  actions. 
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The  intelligent  actions  generated  from  statistical  analyses  of  different  data  sources  can  guide  a more 
efficient  use  of  tools,  processes,  organizations  and  institutional  culture.  In  the  same  way,  academic  analytics 
has  the  potential  to  create  actionable  intelligence  to  improve  teaching,  learning,  and  student  success  to  predict 
which  students  are  in  academic  difficulty  or  focusing  on  specific  learning  needs[4].  Higher  education  has 
traditionally  been  inefficient  in  its  data  use,  often  operating  with  substantial  delays  in  analyzing  readily 
evident  data  and  feedback.  Organizational  processes  often  fail  to  utilize  large  amounts  of  data  on  effective 
learning  practices,  student  profiles,  and  needed  interventions  [7]. 

Something  must  change  to  improve  the  efficiency  and  quality  of  higher  education.  Analytics  in  education 
must  be  transformative,  altering  the  existing  teaching,  learning,  and  assessment  processes,  the  academic 
work,  and  administration  tasks.  Analytics  provides  a new  model  for  university  leaders  to  improve  teaching 
and  learning  processes  and  will  serve  as  a foundation  for  changes.  But  using  analytics  requires  that  we  think 
carefully  about  what  we  need  to  know[5].  Richer  data  sets,  new  ways  of  extracting  and  organizing  data,  more 
sophisticated  predictive  models,  and  additional  research  will  drive  the  evolution  of  analytics. 

One  of  the  in  demand  changes  is  provided  by  the  use  of  Learning  Analytics.  Learning  Analytics  means 
“the  measurement,  collection,  analysis  and  reporting  of  data  about  learners  and  their  contexts,  for  purposes 
of  understanding  and  optimizing  learning  and  the  environments  in  which  it  occurs”[\Q]. 

Learning  Analytics  focus  on  the  learning  process,  using  the  data  provide  by  the  LMS,  in  several  ways: 
analyzing  the  discussion  messages  posted  to  identify  the  student’s  risks,  transforming  the  university  system 
or  the  academic  models  and  giving  recommendations  to  students  to  improve  their  habits. 

This  process  of  learning  analytics  provides  very  useful  information  to  both  teachers  and  students. 
Teachers  will  also  let  them  know,  in  real-time,  performance  of  the  students  and  respond,  immediately,  to  the 
most  urgent,  as  well  as,  plan  better  their  teaching  activities.  To  the  students,  on  the  other  hand,  allows  them  to 
have  knowledge  of  their  progress  and  see  if  their  academic  activity  is  enough  to  achieve  the  raised  objectives. 

It  should  be  noted  that  the  use  of  analytics  to  describe  learning  won't  be  easy,  because  we  are  in  the  early 
stages  of  implementation  and  experimentation.  As  the  practice  of  analytics  will  be  refined,  universities  can 
place  better  information  into  the  hands  of  a greater  number  of  people,  enabling  informed  decision-making. 


4.  A CASE  STUDY 

Therefore,  it  seems  to  be  useful  and  prudent  focus  on  the  realization  of  an  analytical  study,  about  the 
employment  that  teachers  have  made  at  one  of  the  platforms  of  teaching  online.  In  this  case  study  has  been 
taken  into  consideration  the  need  to  evaluate  the  use  of  the  tools  used  in  the  teaching/learning  process  in  an 
interaction  environment  related  to  the  combined  method  of  learning  known  as  "Blended  Learning",  that 
integrates,  in  a balanced  manner,  the  classroom  studies  with  virtual  learning.  The  target  in  this  analytical 
study  is  to  improve  the  prospects  of  the  teachers,  not  only  with  training  actions,  but  also  in  orienting  the  use 
of  the  virtual  educational  tools  most  appropriate,  in  order  to  develop  innovative  educational  strategies. 

4.1  An  Analytical  Practice  in  a Spanish  University  Center 

In  September  2005  the  process  of  implementing  the  Moodle  platform  began  in  Educational  degrees  in  the 
area  of  education  taught  in  a Spanish  university  center  of  studies  of  medium  size,  following  a "Blended 
Learning”  system,  which  develops  an  educational  process  that  adapts  to  the  pace  of  learning  for  the  students 
at  a time  and  in  the  place  you  prefer[3]. 

Using  a rigorous  and  accepted  statistical  model,  on  a representative  sample,  we  have  analyzed  the  use  of 
technological  tools  that  offers  the  platform  Moodle  in  order  to  propose  to  teachers  appropriate  strategies  to 
create  spaces  for  interaction  effective  to  get  the  best  results  in  the  teaching/learning  process. 

From  the  analysis  of  the  subjects  studied  to  obtain  the  teacher’s  degree,  it  is  possible  to  find  out  the  total 
number  of  accesses  made,  during  the  academic  year  2010-2011,  to  each  one  of  the  technological  e-learning 
tools  which  are  in  the  Moodle  platform.  This  total  number  of  accesses  is  shown  in  Figure  1 : 
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As  we  can  see,  in  the  sample  that  has  been  obtained,  there  are  many  different  types  of  tools.  So,  it  is 
necessary  to  establish  an  organization  or  classification  of  the  same  in  categories  to  work  with  an  adequate 
number  of  parameters.  In  response  to  the  use  of  the  tools,  and  to  the  indications  that  it  facilitates  the  own 
platform  Moodle,  we  performed  a classification/organization  into  four  categories  {Storage,  Collaboration, 
Communication  and  Assessment)  that  are  displayed,  then,  in  the  Table  1: 


Table  1.  Classification  of  the  tools  by  tools  groups. 


Classification  Tools  Groups 

Storage 

Label,  Resource,  Upload 

Collaboration 

Forum,  Glossary,  Wiki,  Workshop 

Communication 

Calendar,  Chat,  Journal 

Assessment 

Assignment,  Choice,  Lesson,  Quiz,  Survey 

Once  established  this  classification/organization  in  tools  groups,  an  analysis  has  been  made  by  tools 
groups  to  learn  about  the  average  number  of  accesses  for  each  student  to  the  different  groups  that  were  used 
to  teach  the  curriculum  described  of  the  reference  center.  This  average  number  of  accesses  to  each  student  by 
tools  groups  is  reflected  in  the  Figure  2. 


Figure  2.  Average  number  of  accesses  by  tools  groups. 


According  to  available  data,  the  tools  groups  most  commonly  used  are  those  of  Collaboration,  Storage 
and  Assessment  that  correspond  respectively  with  the  tools  Forum,  Resource  and  Assignment  which  are  most 
commonly  used  as  can  be  seen  in  Figure  1.  On  the  other  hand,  the  Calendar,  Chat  and  Journal  tools, 
belonging  to  the  group  of  Communication  tools,  they  are  hardly  used. 

Conducted  a statistical  analysis  of  Chi-square  with  the  sample  data  for  the  academic  year  2010-2011,  we 
find  that  all  the  groups  of  tools  have  variation.  The  tools  group  that  is  experiencing  a higher  variation  in  the 
average  number  of  accesses  per  student  and  per  tools  group,  is  the  group  of  Collaboration  tools  (Chi-square 
value  199.131). 
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If  it  also  performs  a statistical  analysis  of  the  dependence  or  independence  between  the  different  pairs  of 
groups  of  tools  during  the  period  2010-2011,  it  can  be  concluded  that,  all  pairs  of  groups  of  tools  have 
dependency  between  them.  The  greater  dependence  is  between  the  pair  of  Storage  tools  group  with  the 
Assessment  tools  group  (value  Chi-square  148.01).  In  contrast,  the  less  dependence  is  given  between  the 
Collaborative  tools  group  with  the  Assessment  tools  group  (value  Chi-square  35.08). 

After  the  analytical  process  we  can  see  that  dominate  the  access  to  the  of  Collaboration  and  Storage 
tools.  Otherwise  the  access  to  Assessment  tools  group  always  involves  an  access  to  Storage  tools  group. 


5.  CONCLUSION 

Facing  the  great  amount  of  data  that  we  collect  from  online  learning  the  field  of  education  is  seen  in  the  need 
to  process  all  the  information  available  to  improve  their  benefits.  The  combination  of  Big  Data  with  the 
learning  analytics  are  a good  tool  for  the  processing  of  these  data,  which  will  enable  us  to  guide  the  necessary 
reforms  to  adapt  to  the  new  educational  circumstances,  despite  being  two  areas  of  recent  introduction. 

From  this  study  may  be  inferred  a series  of  information  that  enables  us  to  advise  teachers  to  improve, 
both,  the  learning  process  and  the  teaching  process.  Note,  that  the  most  frequently  used  tools  are  the 
corresponding  to  Collaboration  and  Storage  tools,  which  implies  a large  amount  of  data  when  they  are 
handled  by  both  the  teachers  and  the  students.  These  data  must  be  intelligently  organized  to  obtain  their 
maximum  performance  and  to  get  new  improvements  and  optimization  of  learning  methods. 

Likewise,  it  can  be  inferred  from  the  use  of  Assessment  tools  the  need  for  teachers  to  define  new  forms  of 
evaluation  to  increase  the  accesses  to  these  tools.  Instruments  such  as  the  headings  of  evaluation  can  be  an 
alternative  for  the  improvement  of  this  type  of  tools.  Highlight,  in  addition,  the  need  for  this  improvement 
due  to  the  high  dependence  shown  in  the  survey  with  respect  to  Storage  tools. 

Lately,  higher  education  is  beginning  to  worry  about  how  to  improve  its  learning  methods  and  about  how 
to  increase  its  quality,  finding,  in  the  data  processing  new  technologies  and  the  analytical  methods,  valid 
tools  to  achieve  its  aim. 
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ABSTRACT 

The  advance  of  mobile  industry  and  research  has  expanded  e-learning  in  order  to  support  an  efficient  and  effective 
educational  process.  However,  the  promised  benefits  are  as  much  attractive  as  the  existing  difficulties  and  barriers.  In  this 
paper,  we  intend  to  identify  and  summarize  the  critical  factors  in  mobile  learning  through  a quasi-systematic  review. 
Preliminary  results  are  grouped  on  categories  defined  in  previous  studies  accomplished  by  the  researchers. 
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1.  INTRODUCTION 

The  management  of  information  over  massive  volumes  of  distributed  multimedia  data,  and  the  participative 
and  universal  access  to  knowledge  for  citizens  have  contributed  to  the  Web  development  in  90s,  with  a 
special  interest  in  the  educational  domain  (Santos  et  al.,  2012).  The  goal  is  to  propitiate  an  environment  to 
make  the  Information  and  Communication  Technologies  (ICTs)  closer  to  the  called  “international”  citizens  - 
out  of  the  traditional  physical  barriers  to  the  teaching  and  learning  process.  On  the  other  hand,  the 
establishment  of  the  “Web  Era”  underpinned  “Mobile  Era”  in  2000s.  In  the  beginning,  mobile  devices  were 
focused  on  telecom  domain.  However,  these  devices  have  been  used  as  processor-based  technologies  and 
have  become  a large  part  of  people’s  daily  lives.  Due  to  the  growth  and  practicality  of  smartphones,  these 
were  the  ones  chosen  for  the  development  of  the  mobile  learning  systems  (Pettersson  et  al.,  2010). 

Mobile  learning  is  a type  of  e-learning,  i.e.,  the  use  of  technologies  and  tools  to  facilitate  the  teaching  and 
learning  process  (Girao  et  al.,  2012).  The  difference  is  its  support  by  mobile  devices  as  a complement  to 
distance  education  (Wains  and  Mahmood,  2008).  Thus,  the  mobile  learning  research  explores  issues  related  to 
bringing  education  to  mobile  devices  and  vice  versa  (Pettersson  et  al.,  2010),  and  also  shows  a diverse  field 
with  a plethora  of  theories  and  practices  implemented  in  a variety  of  projects  (Erohberg  et  al.,  2009).  Eor 
example,  on  the  technological  side,  a challenge  consists  in  supporting  devices  with  variable  capabilities,  which 
impacts  learning  activities  themselves  to  provide  a personalized  path  for  every  student,  on  the  application  side. 
Einally,  as  discussed  by  Allan  (2010),  mobility  itself  should  be  investigated,  e.g.,  how  to  manage  and  leverage 
context  changes.  These  challenges  are  beyond  the  traditional  systems  and  devices’  projects  and  designs. 

In  the  Brazilian  case,  for  instance,  the  2010  census  had  shown  194  millions  citizens  and  an  interesting  fact: 
this  number  is  lower  than  the  number  of  mobile  devices  - 202  millions  (Teleco,  2011).  Based  on  the 
governmental  efforts  as  well  as  the  facilities  in  using  mobile  Internet,  the  mobile  learning  can  be  the 
technology  to  support  improvements  in  education.  Our  research  group  has  contributing  to  this  global  (and 
national)  challenge  through  the  mapping  of  bandwidth  world  status  and  an  application  of  mobile  learning  in 
negotiation  domain.  However,  the  promised  benefits  of  mobile  learning  are  as  much  attractive  as  the  existing 
difficulties  and  barriers.  In  this  paper,  we  intend  to  formally  identify  and  summarize  the  critical  factors  in 
mobile  learning  through  a quasi-systematic  review.  Preliminary  results  are  grouped  on  categories  defined  in 
previous  studies  accomplished  by  the  researchers.  This  initial  study  will  base  our  initiatives  in  supporting 
negotiation  learning  through  mobile  and  Web  platforms.  Besides  this  section,  the  paper  is  organized  in  the 
following;  Section  2 presents  the  quasi-systematic  review  and  Section  3 concludes  the  paper. 
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2.  MOBILE  LEARNING:  A QUASI-SYSTEMATIC  REVIEW 

This  section  presents  a quasi-systematic  review  to  formally  identify  and  summarize  the  critical  factors  in 
mobile  learning  through  an  item-based  protocol  as  following.  Using  the  GQM  (Goal/Question/Metric) 
approach  (Basili  et  ah,  1999)  to  define  the  research  focus,  the  approach  goal  is  to  analyze  mobile  learning 
systems  for  the  purpose  of  classifying  with  respect  to  identify  critical  factors  (including  those  related  to  the 
act  of  negotiating)  from  the  Computer  Science  researchers’  point  of  view,  in  the  context  o/each  publication. 
The  research  question  (RQ)  could  be  established  as;  RQ  - What  are  the  critical  factors  in  mobile  learning? 

2.1  Review  Plan 

The  review  steps  can  be  structured  through  a plan  (Biolchini  et  al.,  2007).  The  steps  for  executing  the  quasi- 
systematic  review  were  pointed  out:  (i)  search  string:  define  the  string  based  on  keywords  (or  synonymous) 
obtained  from  PICO  (Population,  Intervention,  Comparison  and  Outcome);  (ii)  source  selection:  define  the 
libraries  where  the  primary  studies  will  be  searched  for;  (iii)  studies  selection : define  the  process  of  selecting 
the  relevant  studies  based  on  quality  criteria;  (iv)  data  extraction:  define  and  extract  which  data  should  be 
collected  from  the  selected  studies  in  order  to  answer  the  RQ;  and  (v)  result  summarization:  analyze  the 
information  and  group  the  studies  aiming  at  effectively  answer  the  RQ. 

In  the  step  (i),  the  populatiou  is  the  group  that  will  be  observed  by  the  intervention;  in  case,  publications 
on  mobile  learning.  The  iuterveutiou  consists  in  what  will  be  observed  in  the  context  of  the  study;  in  case, 
mobile  learning  systems.  The  comparison  focuses  on  analyzing  some  specific  parameters  in  order  to  show 
contrasts  and  other  kinds  of  differences;  in  case,  critical  factors,  including  those  related  to  the  act  of 
negotiating  (this  was  considered  due  to  the  future  work).  The  outcome  is  the  metrics  used  to  measure  the 
effect,  i.e.,  the  types  of  results  expected  in  the  end  of  the  study;  in  case,  it  is  the  same  of  intervention  and  the 
goal  is  to  provide  a classification  based  on  the  critical  factors  found  in  the  included  studies. 

Considering  the  step  (ii),  the  scientific  libraries  selected  for  the  search  were  lEEExplorer^ , Scopus^,  and 
ScienceDirect^ . The  studies  language  should  be  English  and  they  must  be  available  at  Internet,  since  the 
search  is  done  in  Web  engines.  In  turn,  the  step  (iii)  has  established  the  studies  iuclusiou  aud  exclusiou 
criteria;  the  studies  must  present  mobile  learning  systems  and  a discussion  of  critical  factors  (including  those 
related  to  the  act  of  negotiating).  The  procedure  consisted  in:  running  the  search  string  at  the  selected  Web 
engines;  joining  the  abstracts  of  the  initial  set  of  studies  and  reading  them  in  order  to  evaluate  according  the 
mentioned  criteria;  and  reading  the  full  text  for  included  studies. 

The  reasoning  related  to  steps  (i)  and  (ii)  generates  an  initial  search  string  with  many  synonymous  for 
“mobile”,  i.e.,  mobility  and  mobile  device,  as  well  as  for  “systems”,  i.e.,  program,  tool  and  decision  making 
support',  the  word  negotiation  was  included  due  to  the  future  interest  in  using  mobile  leaning  for  negotiation. 
However,  the  number  of  results  was  almost  one  thousand  papers,  because  these  synonymous  had  contributed 
to  retrieve  researches  about  networking  and  communication.  Thus,  the  search  string  was  refined  and  its  final 
structure  is  the  following: 

(negotiation)  AND  ("mobile  learning"  OR  "mobile  ecosystems")  AND  (system  OR  software) 

2.2  Review  Execution 

Since  the  review  plan  was  finished,  the  search  string  was  formally  executed  at  the  selected  Web  engines  and 
the  results  are  shown  in  Table  1.  First  of  all,  234  files  were  originally  retrieved  from  the  Web  engines, 
including  5 repeated  papers.  After  removing  these  papers,  the  set  has  been  reduced  to  199,  which  were 
analyzed  by  the  researchers,  considering  titles  and  abstracts.  75  papers  were  selected  from  this  step  to  read 
the  full  text.  Finally,  28  papers  were  effectively  included  to  the  data  extraction.  So,  the  step  (iv)  described  in 
Section  3.1  was  executed.  The  researchers  focused  on  extracting  the  following  five  types  of  data  previously 
included  in  the  review  plan: 


’ http://ieeexplore.ieee.org 
^ http://www.scopus.com 
^ http://www.sciencedirect.com/ 
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• Year  of  publication; 

• Vehicles  of  publication:  journal  or  event  (i.e.,  proceedings); 

• Focus  of  the  paper:  model  (e.g.,  framework,  architecture  etc.),  tool  (e.g.,  prototype,  infrastructure 
etc.)  or  case  study  (e.g.,  observational  case,  interviews  etc.); 

• Type  of  negotiation  scenario:  business  case  (i.e.,  negotiation  in  a commercial  scenario)  or 
educational  case  (i.e.,  negotiation  in  teaching  and  learning  activities); 

• Critical  factor  for  mobile  learning  (i.e.,  research  contribution). 


Table  1.  Results  from  the  refinement  process 


Web  engines 

Results 

Duplicated  papers 

Analyzed  papers 

Selected  papers 

Included  papers 

1 . ScienceDirect 

49 

0 

2.  Scopus 

60 

5 

199 

75 

28 

3.  lEEExplorer 

125 

30 

Total 

234 

35 

2.3  Data  Extraction  and  Result  Summarization 

From  the  28  papers  included  after  refinement  process,  the  researchers  extracted  the  data  based  on  the  scope, 
as  shown  in  Table  2.  Finally,  the  step  (v)  could  be  completed.  As  the  RQ  requires,  the  Table  2 shows  the 
critical  factors  in  mobile  learning  (last  column).  First  of  all,  (1)  the  papers  were  published  in  the  last  ten 
years,  more  than  a half  of  them  from  201 1 and  in  journal  vehicles.  A reason  for  this  can  be  the  fact  that 
mobile  learning  research  is  growing  based  on  the  establishment  of  the  “Mobile  Era”,  and  many  case  studies 
were  executed  since  it  consists  in  a real  and  large  application  of  recent  Computer  Science  research.  Thus, 
researchers  used  real  scenarios  and  domains  in  order  to  verifying  models  and  tool,  and  also  generating 
indications  and/or  evidences  in  mobile  learning  empirical  research.  Moreover,  the  empirical  basis  can  explain 
the  number  of  papers  related  to  proposition  of  models  such  as  ontologies,  frameworks,  architectures,  context- 
aware  structures,  user  experience  techniques  etc. 


Table  2.  List  of  quasi-systematic  review  results  and  extracted  data 


ID 

Reference 

Year 

Vehicle 

Focus 

Type  of 
Negotiation 

Critical  Factor  in  Mobile  Learning 

1 

Alvarez  et  al. 

2011 

Journal 

Case  Study 

Educational  Case 

Tablets  PCs  strengthen  collective  discourse 

2 

Arroyo  et  al. 

2008 

Journal 

Model 

Educational  Case 

Context-aware  information  based  on  calendar 

3 

Basit  and  Matskin 

2011 

Event 

Model/Tool 

Educational  Case 

User  Model,  User  Experience  and  Places/Spaces 

4 

Bollen  et  al. 

2004 

Event 

Model/Tool 

Educational  Case 

Communication  based  on  sending  short  messages 

5 

Botick  et  al. 

2011 

Journal 

Model 

Business  Case 

Collaboration  and  groups  creation 

6 

Chung 

2012 

Event 

Model/Tool 

Business  Case 

Communication  patterns  and  protocols  (reality) 

7 

Darabant  and  Todoran 

2006 

Journal 

Tool 

Business  Case 

Data  synchronization  using  wireless  technologies 

8 

Fachrunnisa  and  Hussain 

2013 

Journal 

Model/Tool 

Business  Case 

Trust  in  industrial  digital  ecosystems 

9 

Fallahkhair  et  al. 

2005 

Event 

Tool 

Educational  Case 

Television  and  mobile  phone  assisted  language 

10 

Feijo  et  al. 

2009 

Journal 

Model 

Business  Case 

Different  origins/cultures  and  diversity  of  content 

11 

Gerosa  et  al. 

2010 

Journal 

Case  Study 

Educational  Case 

Mobile  notifications  for  better  forums 

12 

Infante  et  al. 

2009 

Journal 

Model/Tool 

Educational  Case 

Single  Display  Groupware  with  Multiple  Mice 

13 

Kallenback  et  al. 

2010 

Event 

Case  Study 

Educational  Case 

Learners'  stress  (heart  rate  and  skin  conductivity) 

14 

Khan  and  Matskin 

2011 

Event 

Model 

Educational  Case 

Spaces/Places  in  multi-agent  systems 

15 

Lan  et  al. 

2011 

Journal 

Model/Tool 

Business  Case 

Imprecision  and  uncertainty  (fuzzy  systems) 

16 

Lan  and  Sie 

2010 

Journal 

Case  Study 

Educational  Case 

Communication  comparison  (SMS,  e-mail,  RSS) 

17 

Lan  et  al. 

2012 

Journal 

Case  Study 

Educational  Case 

Online  asynchronous  discussion  (feedback) 

18 

Li  et  al. 

2011 

Event 

Model 

Educational  Case 

Augmented  reality 

19 

Liu  et  al. 

2007 

Event 

Tool 

Educational  Case 

Shared  display  groupware  (tete-a-tete-oriented) 

20 

Nguyen  and  Pham 

2012 

Event 

Model/Tool 

Business  Case 

Context-aware  mobile  learning  architecture 

21 

Pimmer  et  al. 

2012 

Journal 

Case  Study 

Business  Case 

Integration  to  social  network  sites 

22 

Su  et  al. 

2011 

Journal 

Tool 

Business  Case 

Data  mining  (clustering/decision  tree  approach) 

23 

Trifonova  and  Ronchetti 

2004 

Event 

Model 

Educational  Case 

Context  discovery  and  mobile  content  adaptation 

24 

Tsai  et  al. 

2011 

Journal 

Case  Study 

Business  Case 

Evolve  business  environment  to  be  mobile-based 

25 

Tsai  et  al. 

2012 

Journal 

Case  Study 

Business  Case 

Realistic  and  close-to-real-life  information 

26 

Wang  et  al. 

2002 

Event 

Tool 

Business  Case 

e-marketplace  based  on  agents  and  mobile  agents 

27 

Zhao  et  al. 

2010 

Event 

Model 

Educational  Case 

Learners  get  instant  help  from  other  participants 

28 

Zurita  et  al. 

2007 

Journal 

Model/Tool 

Educational  Case 

Domain  context  and  physical  proximity 
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Second  of  all,  (2)  more  than  a half  of  the  papers  has  focused  on  mobile  learning-based  negotiation 
considers  the  educational  case  - the  actors  are  students  and  teachers,  the  platform  is  a web  and/or  mobile- 
based  infrastructure,  and  the  artifacts  are  the  learning  activities,  contents,  discussions  and  notifications.  This 
indicates  that  negotiation  leaning  should  be  more  explored  in  a business  perspective,  in  specialized  platforms 
and  infrastructures.  Finally,  (3)  the  main  critical  factors  of  mobile  learning  can  be  summarized  in  ten 
categories,  as  shown  in  Table  3.  These  categories  were  elaborated  by  the  researchers  considering  the  studies 
(Rodrigues  et  al.,  2011)  and  (Girao  et  al.,  2012).  The  main  five  categories  are:  human  and  business  aspects, 
medias  integration,  communication,  context-aware  and  collaboration.  However,  it  is  important  to  observe 
that  the  other  categories  correspond  to  new  initiatives  to  explore  mobile  learning  in  negotiation,  considering 
the  new  practical  trends  in  this  decade  (e.g.,  augmented  reality  and  social  networks). 


Table  3.  Categories  of  critical  factors  mobile  learning 


Categories 

List  of  references  (ID) 

Place  and  Space 

3,  14 

Groups  creation 

5 

Medias  integration 

7,9,  12,  19 

Data  mining 

15,22 

Augmented  Reality 

18 

Social  Networks 

21 

Human  and  business  aspects 

8,  10,  13,24,  25,26 

Collaboration 

1,  17,27 

Communication 

4,  6,  11,  16 

Context-aware 

2,  20,  23,  28 

3.  CONCLUSION 

Since  Computer  Science  has  been  established  as  the  “third  pillar”  in  a transition  from  Web  Era  to  Mobile 
Era,  many  approaches  for  mobile  learning  are  emerging  in  the  last  years.  So,  this  paper  formally  identifies 
and  summarizes  the  critical  factors  of  mobile  learning  as  the  main  contribution.  A quasi-systematic  review 
plan  were  established  and  executed  in  order  to  preliminary  discuss  data  extraction  and  results  summarization. 
The  results  were  grouped  based  on  categories  defined  in  previous  studies  accomplished  by  the  researchers. 
As  a future  interest,  we  observed  that  mobile  learning-based  negotiation  is  being  a topic  in  the  last  ten  years, 
especially  from  2009.  Thus,  a deeper  analysis  of  the  results  should  be  done  as  well  as  a complete  systematic 
review  focused  on  Web-  and  Mobile-based  environments  aiming  at  comparing  the  results  with  this  initial 
research  in  mobile.  A list  of  tools  will  be  generated  to  compare  them  considering  the  mentioned  categories  in 
order  to  establish  an  environment  to  supporting  negotiation  learning  based  on  Web  and  mobile  platforms. 
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ABSTRACT 

Development  in  the  smartifacts  and  wireless  network  has  brought  changes  in  the  approaches  and  usages,  as  well  as 
producing  and  sharing  of  the  information.  The  learners  are  changing  into  independent  provider  from  consumer  of 
information.  In  order  to  teach  the  students  effectively  with  this  trend,  changes  in  the  education  is  inevitable.  In  Korea, 
SMART  education  has  been  implemented  to  fit  into  the  trend.  SMART  education  can  be  understood  as  the  change  in  the 
paradigm  which  grafts  newly  developed  technologies  onto  education  to  meet  the  changes  and  the  developments  in  the 
school  systems.  There  are  various  instruction  models  for  ideal  education  in  the  changed  academic  environment.  This 
research  analyzed  the  SMART  education  used  in  Korea,  and  suggests  that  there  is  a need  for  further  research  on  the 
instruction  models. 


KEYWORDS 

Smart  Education,  Instruction  Model. 


1.  INTRODUCTION 

Development  in  the  twenty-first  century  high  technologies  and  information  technologies  has  a great  effect  on 
the  educational  system  today.  Especially,  the  development  of  smartifacts  connected  to  the  mobiles,  the 
ubiquitous  learning  environment  has  been  established  for  learning,  unlimited  of  time  and  place.  Also  the 
strong  networking  function  which  allows  collaboration  allowed  the  various  activities  such  as  participation 
and  sharing.  With  these  functions,  the  students  gained  the  choices  over  various  forms  of  educational  data  and 
information;  they  are  becoming  independent  information  providers.  Also  the  instructors  now  hold  a 
combined  role  as  enforced  facilitator  and  coordinator  who  operates  and  choose  the  information  technologies. 
These  changed  in  the  paradigms  of  educations  need  new  instruction  models.  Therefore,  this  research 
examined  SMART  instruction  models  and  offers  directions  for  further  developments. 


2.  SMART  EDUCATION 

2.1  Definition  and  Characteristic  of  SMART  Education 

The  definition  of  SMART  education  ranges  from  microscopic  perspective  to  macroscopic  perspectives 
depending  on  the  scholars;  education  that  uses  smart  technologies  to  change  in  the  paradigm  of  education  for 
the  future.  However,  the  definition  of  SMART  education  proposed  by  the  Ministry  of  Education,  Science  and 
Technology  (2011)  is  explained  in  terms  of  more  macroscopic  perspectives;  its  purpose  of  overall  changes  in 
the  education  systems.  In  the  other  words,  it  is  defined  “SMART  learning  is  an  intelligent,  tailored 
instruction-learning  supporting  system,  in  which  the  demands  of  the  21st  century  information  technology 
society  are  met  with  changes  in  the  overall  education  system  such  as  pedagogy,  curriculum,  assessment,  and 
teacher.  It  is  a combination  of  human  centered  social  learning  and  adaptive  learning,  based  on  the  best 
network  communication  environment.”  The  Ministry  of  Education,  Science,  and  Technology  explains  the 
SMART  education  with  its  acronym  (Table  1). 
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Table  1.  Characteristic  of  Smart  Education 


Initial 

Feature 

Details 

S (Self-directed  ) 

Knowledge 

manufacturer 

Intelligent 

Change  in  the  role  of  the  student  from  the  consumer  to  the  provider,  role  of  the 
instructor  from  the  knowledge  messenger  to  the  educational  mentor 
A self-conducted  learning  system  with  online  achievement  evaluation  and 
prescription 

M(Motivated) 

Experience- 

centered 

Problem 

Solving 

Emphasize  on  the  experience-centered  learning  method  in  the  standardized 
textbook  based  education 

Aim  for  creative  problem  solving  and  the  process-based  individualized 
evaluation 

A(Adaptive) 

Flexibility 

Enforced  flexibility  of  the  education  system  and  personalized  learning 
experience  based  on  the  preferences  or  future  careers 

Individualizatio 

n 

Role  of  school  changes  from  delivering  mass  knowledge  to  providing 
individualized  learning  with  regards  to  the  level  and  the  aptitude  of  the  student 

R(Resource- 

Open  Market 

Based  on  the  Cloud  educational  service,  various  contents  developed  by  the 
publics,  privates  and  individuals  are  applied  to  the  education 

free) 

Social 

Networking 

Expansion  of  collaborative  learning  using  domestic  and  oversea  learning 
resources  using  collective  intelligence  and  social  learning 

T(Technology 

embedded) 

Open  Education 

An  open  environment  that  offers  desired  learning  experience  regardless  of  time 
and  place,  and  also  guarantees  maximum  learning  options  with  various 
education 

2.2  Purpose  and  Aim  of  SMART  Education 

The  aim  of  SMART  education  is  to  cultivate  the  individuals  into  the  future  global  leaders  using  the 
information  technologies  to  prepare  the  students  with  characteristics  and  abilities  of  twenty-first  century.  In 
order  to  achieve  its  aim,  a total  of  5 different  practical  assignments  stimulate  SMART  education.  Firstly,  in 
the  educational  contents,  a step-wise  expansion  of  digital  text  book  is  pushed  forward  for  effective  learning 
of  students  and  educational  support  of  the  instructors.  Secondly,  in  the  educational  methods,  the  student’s 
learning  experiences  regardless  of  the  geographical  difficulties  are  enforced  through  active  online  courses. 
Thirdly,  in  the  educational  environment,  it  creates  safe  donation  and  sharing  of  education  contents.  Fourthly, 
in  the  capability  of  the  facilitators,  it  offers  various  training  opportunities  and  programs  for  the  facilitators  to 
strengthen  the  instructing  abilities  for  the  SMART  education.  Lastly,  in  the  foundation  aspects.  Cloud 
education  service  environment  is  achieved  for  easy  and  unlimited  access  to  the  educational  contents  for  the 
instructors  and  students. 


3.  EXAMPLES  AND  ANALYSIS  OE  SMART  COURSES 

3.1  Smart  Instruction  Model 

3.1.1  Idea  Sharing  Instruction  Model 

‘Idea  Sharing’  instruction  model  is  a class  model  in  which  the  students  first  independently  think  about  the 
proposed  problem  then  shares  the  ideas  with  fellow  classmates  and  the  instructor  using  the  smartifacts  to 
examine  and  reshape  the  solutions.  This  instruction  model  is  listed  step-wise  in  Table  2 
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Table  2.  Idea  Sharing  Instruction  model 

Steps 

Activities 

Come  up  with  Ideas 
Sharing  Ideas 
Combining  Ideas 

Form  individual  ideas,  write  down  the  ideas,  finalize  the  ideas  to  be  shared  with  class 
Save  and  share  the  ideas,  examine  ideas 
Complement  individual  ideas,  confirm  the  changes 

Looking  at  the  detailed  steps  of  the  instruction,  the  first  “come  up  with  ideas”  step  is  a prior  knowledge 
based  brainstorming  step  which  requires  the  students  to  come  up  with  and  express  individual  ideas.  The 
second  step,  “Sharing  Ideas”,  focuses  on  the  main  activity  of  sharing,  saving,  and  examining  the  ideas  (such 
as  concepts,  processes,  opinions,  etc.)  with  everyone  else  on  the  network.  Finally,  “Combining  Ideas”  step 
offers  the  opportunities  to  examine  other  individuals’  ideas  and  complementing  one’s  own  ideas,  as  well  as 
extra  information  searching  opportunities  if  needed. 

3.1.2  Research  Activities  Centered  Instruction  Model 

‘Research  Activities  Centered’  instruction  model  is  based  on  the  research  activities  in  which  problem  is 
solved  through  following  steps;  gathering  information  on  the  internet  or  database,  critical  analysis,  and 
rational  decision  making.  The  step-wise  instruction  model  is  shown  in  Table  3. 


Table  3.  Research  Activities  Centered  Instruction  Model 


Steps 

Activities 

Planning 

Search  and  Gathering 
Presentation 

Create  a question  relative  to  the  search  (or  provided  by  the  instructor).  Consider 
how  to  conduct  the  investigation 

Collect  the  information  by  searching  the  internet  or  database,  confirm  the  credibility 
of  the  information/data,  sort  and  analyze  collected  information 
Present  the  results,  give  feedback,  and  reflect 

Looking  into  the  detailed  instruction  steps,  the  first  “Planning”  step  heavily  weighs  creating  a relative 
question  to  the  research  topic  in  order  to  decide  what  to  search  for.  The  second  step,  “Search  and  Gathering”, 
is  a step  in  which  information  credibility  is  decided  upon  searching  the  internet  and  the  database,  and 
collected  information/data  is  sorted  and  analyzed  accordingly.  In  the  last  step,  “Presentation,”  a simple, 
to-the-point,  accurate  presentation  that  answers  the  research  topic  prepared  by  the  instructor  is  required. 

3.1.3  SMART  Activity  Based  Instruction  Model 

SMART  activity  based  Instruction  Model  is  not  limited  to  the  details  of  the  previous  instruction  models;  it  is 
based  on  the  various  SMART  activities  that  can  expedite  the  capabilities  of  the  21  century  students.  The 
detailed  instruction  model  is  shown  in  Table  4. 


Steps 

Introduction 

Activities 

Conclusion 


Table  4.  SMART  activity  based  Instruction  Model 
Activities 

Have  a critical  mind,  problem  deduction 

SMART  activities  using  various  academic  supporting  equipment  to  expedite  the  21 
century  academic  capabilities 

Feedback  and  Evaluation  on  the  activities,  formative  evaluation  on  the  academic 
activities 


In  the  “Activities”  step,  following  SMART  activities  are  implemented  to  accelerate  the  capabilities  of  21 
century  learning.  (Table  5) 
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Table  5.  SMART  activity  and  Use  of  Supplementary  Equipment 


Activities 

Use  of  Supplementary  Equipment 

Collaboration 

Writing  a scenario  or  group  report  using  Google  Drive 
Presentation  after  group  work  using  Prezi 

Communication 

Sharing  opinions,  and  rewriting  through  SNS(Twitter,  Facebook) 
Group  discussion  and  Instructor  consultations  using  Google  Hang-out 

Thinking 

Reconstitution  of  the  data  using  one-note,  ever-note 
Drawing  mind  map  using  x-mind 

Creativity 

Brain  Writing  using  Sticky 

Producing  creative  animations  using  StopMotion,  Animating  Touch 

Research  and  Experience 

Research  activities  using  the  map  and  measurement  apps 

3.1.4  Analysis  of  SMART  Instruction  Model 

First,  the  ‘Idea  Sharing’  instruction  model  aims  to  provide  activities  such  as  participation  and  sharing  using 
the  smartifacts.  However,  these  activities  are  available  with  previous  classroom  environment  through 
presentations  or  discussions.  On  the  other  hand,  there  is  a risk  of  creating  a ‘quiet  classroom’  with  the  use  of 
smartifacts.  Therefore,  ‘Idea  Sharing’  instruction  model  can  be  further  improved  by  having  the  students  to 
prepare  to  share  the  newly  gained  knowledge  or  ideas  in  the  class  for  the  following  class  period. 

Secondly,  ‘Research  activity  centered’  instruction  model  has  merit  of  searching  and  using  the  information 
real  time.  However,  it  depends  more  on  the  internet  and  database  search  rather  than  on  the  students’  critical 
thinking  development;  also  time  can  be  wasted  focusing  on  putting  presentation  together  using  the  computer, 
rather  than  being  spent  on  the  academic  contents.  Therefore,  it  is  important  to  secure  enough  time  to  analyze 
and  communicate  the  gathered  information  and  spend  less  time  on  the  search  and  gathering. 

Thirdly,  ‘SMART  activity  centered’  instruction  model  allows  the  students  to  present  the  results  in  various 
forms  such  as  group  report  and  animation,  which  was  harder  to  do  in  previous  classroom  environments. 
However,  when  the  focus  is  on  the  activities,  the  students  may  be  distracted  using  the  applications  causing 
less  academic  achievement.  Therefore,  it  is  important  to  induce  the  students  to  have  more  interest  in  the 
academic  evaluation  or  feedbacks  rather  than  in  the  activities. 

4.  CONCLUSION  AND  RECOMMENDATION 

SMART  education  can  be  seen  as  the  change  in  the  paradigm  which  brings  in  the  smartifacts  of  the  twenty- 
first  century  digital  media  environment  to  offer  activities  such  as  search,  sharing,  communication  and 
application  to  improve  classroom  experiences.  Various  instruction  models  are  being  developed  with  SMART 
education  introduced,  and  analyzing  this  trend,  it  was  found  that  SMART  education  allowed  various  real  time 
data  search,  communication,  and  presentation  to  be  possible;  with  previous  instruction  models,  such  activities 
were  not  easily  granted.  However,  if  the  spotlight  is  on  the  usage  of  smartifacts  in  the  activities,  the  SMART 
education  will  not  be  practical.  Therefore,  there  is  a need  for  continuous  research  on  the  instruction  models  to 
increase  the  students’  capabilities  and  better  class  instruction  methods.  Clearly  indicate  advantages, 
limitations  and  possible  applications. 
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ABSTRACT 

The  History  Harvest  project  (http://historyharvest.unl.edu)  is  an  open,  digital  archive  of  historical  artifacts  gathered  from 
communities  across  the  United  States.  Each  year,  The  University  of  Nebraska-Lincoln  Department  of  History  partners 
with  local  institutions  and  community  members  within  a highlighted  area  to  collect,  preserve,  and  share  their  rich,  but 
often  hidden,  histories.  Advanced  undergraduates,  working  as  a team  and  with  the  guidance  of  faculty  members  and 
graduate  students,  "harvest,"  digitize,  and  curate  the  artifacts  and  stories  they  collect.  The  History  Harvest  project  is 
rooted  in  the  belief  that  our  collective  history  is  more  diverse  and  multi-faceted  than  most  people  give  credit  for  and  that 
most  of  this  history  is  not  found  in  archives,  historical  societies,  museums  or  libraries,  but  rather  in  the  stories  that 
ordinary  people  have  to  tell  from  their  own  experience  and  in  the  things  - the  objects  and  artifacts  - that  they  keep  and 
collect  to  tell  the  story  of  their  lives.  The  History  Harvest,  then,  affirms  the  importance  of  local  people,  local  communities 
and  everyday  experience  in  the  broader  narrative  of  American  history  by  providing  an  innovative  opportunity  for 
ordinary  people  to  share  their  historical  artifacts,  and  their  stories,  for  inclusion  in  a unique  digital  archive  of  what  we  are 
calling  the  "people's  history."  This  new  public  resource  is  then  available  for  educators,  students  and  anyone  else 
interested  in  engaging  U.S.  history  from  this  more  democratic,  or  grassroots,  perspective.  This  short  paper  on  our  work- 
in-progress  examines  the  experiential  learning  basis  for  The  History  Harvest  project  and  its  rationale  for  democratizing 
history  in  a digital  age. 


KEYWORDS 

History,  Experiential  Learning,  Authentic  Learning,  Digital  History,  Digital  Humanities 


1.  INTRODUCTION 

In  The  History  Harvest,  an  experiential  learning  initiative  led  and  organized  by  University  of  Nebraska- 
Lincoln  to  collect,  preserve,  and  digitize  the  people's  history,  we  propose  a community-based  model  for 
experiential  learning  in  the  undergraduate  history  curriculum.  In  this  short  paper  we  provide  the  rationale  and 
background  for  our  work-in-progress.  The  project  has  been  featured  recently  in  The  Chronicle  of  Higher 
Education  (December  2012)  and  the  American  Historical  Association's  magazine  Perspectives  on  History 
(January  2013).  Overall,  the  History  Harvest  project  aims  to  create  a new  model  of  undergraduate  learning  in 
the  humanities  that  is  team-oriented,  student-led,  community-based  and  integrated  through  digital 
technologies.  Furthermore,  the  History  Harvest  project  seeks  to  democratize  history  by  creating  public 
conversation  about  the  materials  of  history  and  their  meaning,  as  well  as  through  the  creation  of  an  online 
public  archive,  which  makes  available  new  material  — the  artifacts  of  everyday  life  — for  historical  study,  K- 
12  instruction,  and  life-long  learning.  To  date,  we  have  run  four  "harvests"  with  both  undergraduate  and 
graduate  students  in  2010-2012. 
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2.  MODELING  EXPERIENTIAL  LEARNING  IN  DIGITAL  HISTORY 

Grounded  in  the  pedagogy  of  experiential  learning,  The  History  Harvest  places  advanced  undergraduate 
students  in  a class  designed  to  plan,  run,  and  execute  an  on-site  community  digitization  effort.  Students  work 
with  and  within  the  community,  undertake  interpretive  analysis  of  the  materials  harvested,  and  curate  these 
objects  in  a web  site  archive  using  widely  accepted  metadata  standards.  Led  by  faculty  experts  and  supported 
by  graduate  students,  undergraduates  read  secondary  works  about  the  major  historical  subjects  relevant  to  the 
community's  history,  and  they  continually  reflect  on  the  process  of  doing  history  as  they  work  through  the 
steps  of  planning,  organizing  and  executing  the  History  Harvest  event,  then  processing  and  archiving 
harvested  materials.  In  addition,  each  "harvest"  seeks  out  opportunities  for  students  to  work  with  local 
historical  societies,  museums,  schools  and  other  organizations  to  further  the  broad  goals  of  the  project, 
support  community-based  historical  organizations  and  generally  cultivate  a broader  and  more  sustained 
interest  in  historical  preservation  and  knowledge.  For  instance,  students  worked  with  the  Great  Plains  Black 
History  Museum  in  North  Omaha  to  rehabilitate  important  archival  materials  that  had  been  damaged  and  the 
Making  Invisible  Histories  Visible  summer  history  program  for  "at-risk"  middle  school  and  early  secondary 
school  students.  (Teaching  Tolerance  2012,  Jonassen  et  al.  2003). 

The  benefits  of  experiential  and  authentic  learning  seem  clear  to  researchers.  Marylin  Lombardi  has 
defined  "authentic  learning"  as  focused  on  "real-world,  complex  problems  and  their  solutions  using  role- 
playing  exercises,  problem-based  activities,  case  studies,  and  participation  in  virtual  communities  of 
practice."  Students  immersed  in  these  activities  cultivate  important  skills,  including  "synthetic  ability"  to 
recognize  patterns,  "patience"  to  follow  long  arguments,  and  "flexibility"  to  work  across  disciplinary  and 
cultural  boundaries.  (Lombardi,  2007)  Although  aimed  primarily  at  K-12  teaching  and  learning,  David  H. 
Jonassen's  comprehensive  research  on  experiential,  constructivist  pedagogy  stresses  the  importance  of 
uncertainty  and  complexity  in  learning  scenarios.  (Jonassen,  2005)  He  emphasizes  the  concept  of  computers 
as  "mindtools"  or  "cognitive  tools"  and  through  which  students  develop  "multiple  mental  representations, 
including  structural  knowledge,  procedural  knowledge,  reflective  knowledge,  images  and  metaphors  of  the 
system,  of  strategic  knowledge  as  well  as  social/relational  knowledge,  conversational/discursive  knowledge 
and  artifactual  knowledge."  (Jonassen,  2011). 

But  historians  have  been  slow  to  adopt  such  pedagogies,  despite  the  opportunities  that  digital 
technologies  afford.  In  an  essay  on  the  future  of  digital  history  and  undergraduate  education,  Edward  L. 
Ayers  recently  called  for  students  to  participate  in  a cycle  of  "generative  scholarship."  (Ayers,  2013)  He 
suggested  that  students  build  their  work  alongside  ongoing  research  projects  so  that  their  contributions  are 
assessed,  validated,  and  preserved.  The  fundamental  question  humanities  faculty  face  is:  how  do  we 
reconstitute  our  classroom  practices— literally  what  students  do— for  the  digital  age? 

A number  of  digital  humanities  scholars  and  leaders,  including  Ayers,  have  suggested  that  we  need  to 
experiment  with  new  models  of  undergraduate  education  in  the  humanities.  Founder  of  the  Perseus  Digital 
Library,  Greg  Crane  has  recently  drawn  attention  to  the  need  for  "a  new  culture  of  learning"  not  only  for  the 
field  of  classics,  but  also  more  broadly  for  the  humanities.  According  to  Crane,  "we  need  to  engage  our 
students  and  our  fellow  citizens  as  collaborators.  We  need  a laboratory  culture  where  student  researchers 
make  tangible  contributions  and  conduct  significant  research."  Crane  argues,  "the  crush  of  data  challenges  us 
to  realize  higher  ideals  and  to  create  a global,  decentralized  intellectual  community  where  experts  serve  the 
common  understanding  of  humanity."  (Crane,  2012b). 

Crane's  critique  of  the  humanities  deserves  quoting  at  length:  "Defenders  of  the  humanities  claim  a 
special  role  in  training  citizens  for  a democratic  society  and  often  have  deeply  felt  convictions  about 
democratizing  knowledge  and  including  new  voices.  The  mainstream  of  humanities  research  has,  however, 
focused  upon  virtuoso  scholarship,  published  in  subscription  publications  to  which  only  academics  have 
access,  and  composed  for  small  networks  of  specialists  who  write  letters  for  tenure  and  promotion  and  who 
meet  each  other  for  lectures  and  professional  gatherings.  Students  in  the  humanities  remain,  to  a very  large 
degree,  subjects  of  a bureaucracy  of  information,  where  they  have  no  independent  voice  and  where  they 
never  move  beyond  achieving  goals  narrowly  defined  by  others."  (Crane,  2012a,  Crane  2012b). 

The  History  Harvest  project  is  one  effective  model  to  achieve  these  ends.  In  the  project,  we  attempt  to 
reorganize  the  learning  process  for  undergraduate  students  and  to  provide  standing  for  community  members 
to  have  a voice  in  what  Crane  calls  "the  republic  of  learning."  The  lesson  in  almost  every  digital  history 
project  has  been  that  people  have  materials  to  contribute,  and  they  also  have  expertise.  At  each  "harvest," 
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community-members  are  invited  to  bring  and  share  their  letters,  photographs,  objects  and  stories,  and 
participate  in  a conversation  about  the  significance  and  meaning  of  their  materials.  Each  artifact  and  the  story 
told  about  it  are  digitally  captured  and  then  shared  in  this  free  web-based  archive  for  general  educational  use 
and  study.  Caring  about  everyday  people’s  histories  - their  stories  and  their  objects  - validates  their 
experience,  which,  in  turn,  has  often  resulted  in  an  even  greater  appreciation  of,  and  engagement  with, 
history.  (Crane  2012b). 

The  History  Harvest  on-site  event  features  student-run  digital  imaging  and  filming  tables  for  documents, 
letters,  diaries,  photographs,  art,  and  3-dimensional  objects.  To  date,  community  members  have  shared 
remarkable  documents  and  objects  including:  homestead  family  letter  collections,  railroad  timetables.  Civil 
War  letters,  rare  commemorative  silver  sets,  slave-owned  cups  and  coins,  church  records,  business  records, 
rare  music  sheets,  broadsides,  pamphlets,  uniforms,  photographs,  and  posters.  As  we  turn  to  the  largely 
invisible  archive  in  family  and  community  collections,  we  see  not  only  an  opportunity  to  supplement  our 
other  archival  records,  but  also  to  connect  what  have  been  long  separated  domains  of  historical 
understanding.  Over  time,  as  the  digital  archive  grows  and  more  people  in  different  locales  conduct  harvests, 
exciting  new  opportunities  will  emerge  to  work  with  these  materials  and  create  new  resources  and  tools  for 
historical  exploration,  from  curriculum,  to  audio  and  video  programs,  to  networked  exhibits  that  link 
materials  across  the  many  harvests. 

In  the  1990s  the  animating  spirit  behind  much  of  the  work  in  digital  humanities  was  democratization.  At 
that  time,  a small  group  of  like-minded  librarians,  scholars,  technology  professionals,  and  students  saw  early 
on  that  the  World  Wide  Web  opened  up  new  possibilities  for  scholars  to  communicate  not  only  with  one 
another  but  also  with  the  public,  with  an  audience  largely  unmediated  by  traditional  gatekeepers.  Their 
ambitions  then  were  to  transform  the  way  history  was  understood  by  changing  the  way  it  was  produced  and 
accessed.  In  fact,  we  cannot  change  the  way  history  is  understood  without  changing  the  way  it  is  produced 
and  accessed.  The  History  Harvest  project  seeks  to  recover  this  animating  spirit  in  a time  of  increased 
privatization  and  commercialization  of  the  sources  necessary  to  do  history  and  to  make  good  on  the  promise, 
long  articulated,  but  rarely  achieved,  that  digital  technologies  might  yet  transform  the  way  history,  as  well  as 
the  broader  humanities,  is  taught  and  learned.  It  is  precisely  the  hybrid  formulation  of  this  approach,  blending 
elements  of  traditional  classroom-based  teaching,  with  more  recent  calls  for  innovative,  student-led  and 
community-oriented  approaches,  along  with  digital  and  other  technology-based  methodologies,  that  has 
proven  so  powerfully  transformative  for  teachers,  students  and  local  communities. 

Finally,  as  communities  explore  their  common  heritage,  students  and  participants  recognize  the  real 
consequences  of  history  for  today.  Some  communities  have  had  their  histories  expropriated  and  abused  for 
generations  by  state  powers,  large  institutions,  including  universities,  and  political  organizations;  distorted  by 
mainstream  media  outlets;  or  simply  ignored  or  minimized  by  the  majority.  The  History  Harvest  seeks  to 
address  these  deficiencies  in  the  mainstream  historical  record  by  empowering  communities,  encouraging 
dialogue,  and  enabling  preservation  without  appropriating  the  past  or  taking  material  objects. 

In  the  next  phase  of  the  project  we  will  run  distributed  History  Harvest  classes  at  the  same  time  at  three 
institutions.  Students  in  the  three  sites  will  build  the  digital  archive,  and  work  across  institutions  on 
interpretation,  curation,  and  analysis  of  sources  related  to  a particular  theme.  We  will  conduct  research  to 
evaluate  the  way  History  students  can  be  trained  to  work  with  technical  scripting  and  the  way  Computer 
Science  students  can  be  trained  to  evaluate  and  interpret  data  in  historical  context.  This  interdisciplinary 
nexus  of  skills,  habits  of  mind,  and  communication  lies  at  the  core  of  several  unanswered  research  questions. 
How  do  students  effectively  collaborate  across  disciplinary  boundaries?  What  sequences  of  skill  and  habits  of 
mind  work  most  effectively  in  each  domain?  What  technical  scaffolding  is  necessary  to  support  such  work? 
Using  social  media,  high-speed  Internet  connectivity,  mobile  devices,  and  social  networking  tools,  we 
anticipate  a high  rate  of  adoption  and  adaptation  of  The  History  Harvest  concept.  As  the  project  grows,  the 
interactive,  collaborative  possibilities  are  virtually  limitless. 
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3.  RESULTS  AND  SUMMARY 

Our  results,  measured  qualitatively  with  student  exit  interviews  and  community  participant  responses, 
indicate  the  following:  that  students  undergo  a profound  transformation  in  their  understanding  of  history  and 
its  meaning,  and  develop  a range  of  core  skills  along  the  way,  which  are  applicable  within  a range  of 
potential  future  professional  pathways;  that  there  is  strong  and  enthusiastic  support  from  most  local  people 
and  community  organizations  for  this  sort  of  innovative  "people’s  history"  effort  and  that  the  History  Harvest 
project  often  engenders  broader  local  discussions  among  community  partners  about  the  value  and  meaning  of 
their  history;  and  that  digital  cameras,  scanners  and  other  similar  technologies  allow  for  rapid,  accurate,  and 
efficient  capture  of  historical  materials  on  site,  while  the  online  environment  provides  a new  and  widely 
accessible  way  to  organize  and  share  this  "people’s  history."  (see  http://historyharvest.unl.edu/multimedia). 

Through  this  experiential  learning  project  we  have  the  opportunity  to  make  previously  inaccessible 
materials  visible  and  usable.  Just  as  important,  we  see  the  History  Harvest  as  a way  to  open  up  and  make 
more  accessible  who  is  included  in  our  history.  Alongside  this  ambitious  goal  we  propose  reorganizing  the 
undergraduate  experience  in  the  humanities  by  building  a collective  research  project  for  undergraduate 
contribution.  Taking  advantage  of  the  new  technologies  for  digitization  and  new  media  for  the  presentation 
and  analysis  of  historical  sources.  The  History  Harvest  serves  as  a prototype  for  a model  of  undergraduate  e- 
leaming  in  the  humanities:  one  that  is  experiential,  student-led,  team-oriented,  networked,  community-based, 
"generative,"  and  interdisciplinary. 
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ABSTRACT 

The  Internet  has  brought  about,  a revolution  in  education,  more  precisely  with  regards  to  online  learning.  Online 
education  has  helped  remove  many  barriers  which  traditionally  hindered  access  to  education  due  to  its  relatively  low 
price  and  high  flexibility  in  how  and  when  courses  and  paces  courses  are  taught. 

In  this  study,  we  carry  out  a survey  of  the  need  of  e-leaming  for  students  at  the  Mongolian  University  of  Science  and 
Technology  to  find  out  about  their  learning  preferences. 

E-learning,  if  used  effectively  can  help  address  the  many  short-comings  of  the  traditional  education  methods  as  well  as 
the  inherent  problems  the  classroom  teacher  is  faced  on  a daily  basis  with  a classroom  of  learners  with  different  learning. 
We  therefore  propose  a model  for  e-learning  that  provides  adaptive  tutoring  using  technology-enhanced  learning.  An 
important  aspect  of  Learning  Management  Systems  is  their  ability  to  offer  collaboration  tools  to  build  a community  of 
learners. 

The  aim  of  this  document  is  to  provide  support  for  the  cost-effective  use  of  e-leaming  technology  and  content  in 
education  and  training  by  setting  out  a national  framework  for  the  education  system. 


KEYWORDS 

e-learning,  challenges  of  e-learning,  readiness  of  students  for  e-leaming,  an  e-leaming  framework 


1.  INTRODUCTION 

Mongolian  higher  education  system  currently  has  101  universities,  15  state  universities  and  81  private  own 
universities.  Every  year,  about  a thousands  of  students  apply  to  enroll  into  these  universities  now  the  number 
of  students  are  173  000  [1]. 

Over  the  past  10  years  the  number  of  higher  education  institutions  has  increased  to  178,  of  which  135  are 
private.  Enrollment  at  private  higher  education  institutions  has  more  than  doubled  over  the  past  4 years. 
According  to  data  provided  by  the  Ministry  of  Education,  Culture  and  Science  of  Mongolia,  the  national 
education  system  benefited  US$110,000,000  in  grants  during  the  period  of  1997-2008  and  US$55,000,000 
in  loans  for  1997-2011. 


2.  THE  CURRENT  ISSUES  OE  E-LEARNING  IN  MONGOLIA 

E-Learning  in  Mongolia  began  in  the  late  1990s  and  since  that  time  has  accumulated  a lot  of  experience. 
Several  projects  developing  e-learning  in  Mongolia  were  implemented  with  support  from  UNESCO  in  1992 
- 1997,  1997-2001. 

Given  Mongolia’s  relatively  small  population  of  2.7  million  people  and  with  US$  165,000,000  in  external 
aid,  the  national  authorities  should  be  able  to  ensure  adequate  learning  opportunities  for  all  citizens. 
However,  during  a Conference  of  educational  methodologists  from  remote  aimags  (provinces)  raised 
concerns  about  the  lack  of  text  books,  especially  those  required  in  minority  languages,  lack  of  infrastructure 
and  pointed  out  discrepancies  between  the  investment  in  high-tech,  consultant  services  and  the  outcomes  at 
the  grassroots  level. 
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The  National  Center  for  Non-formal  Distance  Education  had  an  estimated  value  of  US$300  million  from 
the  central  budget  from  2002-2010. 

There  was  no  meaningful  link  found  between  schools  and  industry,  nor  was  there  adequate  financial 
support  in  place  for  the  implementation  of  e-Leaming  systems,  simply  because  many  universities  failed  to 
allocate  sufficient  funds  for  e-Learning. 

E-learning  initiatives  in  Mongolia  are  being  undertaken  mainly  by  universities,  colleges  and  business 
enterprises.  The  main  players  of  e-learning  are  private  and  public  institutions  of  higher  education  as  well  as 
local  and  multinational  corporations. 

The  main  challenges  we  are  currently  facing  are: 

• Lack  of  specialists 

• Insufficient  financial  aid 

• No  standards 

• Incomplete  common  learning  management  system 

• Insufficient  preparation  of  e-tutors.  There  is  a strong  requirement  to  develop  e-Ieaming  framework 
suitable  for  specific  country  to  solve  the  problems.  E-Iearning  frameworks  are  developed  by  a number 
of  specialists  adapting  their  country’s  condition. 

The  main  objective  of  our  research  is  to  propose  an  e-Iearning  framework  for  Mongolian  higher  education 
system  according  to  the  universities  based  on  the  e-Iearning  development  and  implementation  training 
program  developed  by  the  German  International  organization  GIZ  and  Inwent. 

In  this  paper,  we  carry  out  surveys  with  students  at  the  10  different  branch  schools  of  Mongolian 
University  of  Science  and  Technology  to  find  out  about  their  readiness  for  e-Iearning  according  their  own 
preparation,  and  their  perceptual  styles  that  help  them  learn  best. 

700  students  responded  to  the  survey  over  a two-week  period  at  the  end  of  May  2011  from  10  branch 
universities. 

U‘  grade  students  - 231,  second  grade  -166,  3'^"'  grade  183,  4*  grade  - 93,  5*  grade  - 21 

Students  have  permanent  internet  connection  made  up  79.2%  of  the  total  respondents.  Of  these,  the 
different  types  of  connection  available  to  students  were;  home  42%,  university  19%,  internet  cafe  32%. 

48%  of  respondents  were  aware  that  it  is  now  possible  to  use  a internet  connection  1-2  hour  a day,  24%  of 
them  use  3-4  hour  a day. 


Which  subject  will  you  learn  online? 

History  and  social  study-190,  Computer-  284,  Physics  and  mathematics- 85,  Special  course-252,  Foreign  language- 
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Strongly 

disagree 

2 

3 

4 

5 

Strongly 

agree 

I think  I would  be  able  to  read  and  learn,  or  follow  the 
direction  on  a computer  screen  to  accomplish  a task 

56 

61 

96 

111 

279 

5 

I would  able  to  take  notes  while  watching  a video  on  the 
computer  and  would  able  to  have  read  online  or  in  books 

28 

40 

127 

135 

257 

2 

I think  that  I would  be  comfortable  having  several 
discussions  taking  place  in  the  same  online  chat  even 
though  I may  not  be  participating  in  all  of  them 

31 

65 

106 

148 

231 

1 

I think  that  I would  be  able  to  use  online  tools  (e.g  email, 
chat)  to  work  on  assignments  with  learners  who  are  in 
different  time  zones 

36 

49 

125 

156 

204 

1 

I would  rather  listen  to  a lecture  than  read  the  material 
from  a computer  screen 

38 

44 

98 

68 

133 

2 

I would  rather  find  out  information  using  a computer  than 
from  a teacher  or  lecturer 

74 

94 

72 

74 

59 

2 

I cannot  learn  using  only  computers,  I need  the 
teacher/student  contact 

49 

58 

95 

107 

269 

4 

Would  you  like  to  learn  online  instead  of  face  to  face  training  via  the  internet? 


I will  study 

I think  face  to  face 

Td  like,  but  I 

Td  like,  but  I 

I will  not  attend 

I will  not  attend 

online  if  it 

learning  would  be 

am  not  able  to 

don’t  have  own 

online  training 

online  training 

available 

rather  than  online 

use  internet 

computer 

because  of  weak 

because  of  weak 

everyday 

computer  literacy 

usage  internet 

318 

179 

143 

82 

24 

22 
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What  kind  of  learning  materials  do  you  prefer? 


Learning  material  Strongly  disagree 

2 

3 

4 

5 

Strongly  agree 

Printable  book,  learning  materials 

22 

61 

73 

61 

120 

257 

CD,  DVD  materials 

74 

60 

84 

105 

87 

113 

Handbooks  and  reading  books 

28 

40 

83 

86 

120 

192 

Downloadable  materials 

62 

48 

72 

72 

70 

238 

Do  you  think  the  lecture  is  most  important  tool  of  training? 

Training  tools 

Strongly  disagree 

2 

3 

4 

5 

Strongly  agree 

I prefer  the  online  training  rather  than  lecture 

153 

65 

74 

88 

47 

101 

Lecture  is  very  important  tool  of  training 

19 

38 

75 

74 

92 

238 

Lecture  would  be  efficient  with  workbooks 

26 

33 

76 

88 

103 

184 

I think  the  blended  learning  would  be  efficient 

35 

34 

79 

65 

97 

213 

Do  you  complete  the  task  on  time  during  face  to  face  learning?  Yes  -165,  sometimes  - 449,  no  - 65 


3.  A FRAMEWORK  FOR  E-LEARNING  DEVELOPMENT  AND 
IMPLEMENTATION 

In  recent  years  there  has  been  significant  progress  in  the  development  of  technology  to  support  education  and 
training.  This  has  included  not  only  the  development  of  simulators  and  interactive  e-learning  products  to 
support  learning,  but  also  new  learning  management  tools  to  test,  record  and  report  individual  progress. 

The  purpose  of  the  e-Learning  Framework  is  to  help  organizations  to  take  the  next  steps  in  developing 
e-learning.  The  adoption  of  e-learning  is  a key  element  in  improving  the  delivery  of  quality  education. 

The  aim  of  this  paper  is  to  provide  support  for  the  cost-effective  use  of  e-learning  technology  and  content 
in  education  and  training  by  setting  out  a national  framework  for  the  education  system.  The  e-learning 
Framework  for  education  sets  out: 

• The  national  vision  for  the  development  of  e-learning; 

• How  users  can  find  details  of  current  products  and  those  which  are  being  developed; 

• How  organizations,  professional  bodies  and  individuals  can  participate  in  identifying  future 
priorities  for  product  development;  and 

• The  roles  and  organizations  at  different  levels  of  the  education  which  play  a role  in  supporting  the 
development  of  e-learning. 

The  Eight  component  framework  for  e-learning  is  global  instructional  design  method  designed  by  Badrul 
H.  Khan.  This  framework  is  more  complex,  convenient  for  whole  education  system. 

Design,  development,  implementation  and  evaluation  of  open  and  distributed  learning  systems  require 
thoughtful  analysis  and  investigation  of  how  to  use  the  attributes  and  resources  of  the  Internet  and  digital 
technologies  in  concert  with  instructional  design  principles  and  issues  important  to  various  dimensions  of 
online  learning  environments. 

3.1  Stage  1 

E-Learning  management  refers  to  the  management  of  eLearning  projects.  Erom  a business  studies  point  of 
view,  it  refers  to  guidance  and  control.  As  for  eLearning,  ‘management’  is  mostly  concerned  with  the 
planning,  implementation  and  assessment  of  eLearning  projects. 

This  statement,  one  of  the  core  features  of  a project  as  it  states  the  importance  in  project  management  of 
defining  a clear  objective.  It  is  paramount,  however,  that  the  overall  goal  is  clearly  defined  and 
communicated  to  everybody  from  the  very  beginning. 

3.2  Stage  2 

In  this  stage  the  necessary  steps  to  draft  the  concept  for  a course  from  vaguely  formulated  educational  needs 
to  be  developed.. 

The  process  of  instructional  design  should  result  in  a course  that  enables  the  learner  to  reach  a certain 
objective.  Methods  and  media  have  to  be  planned  to  support  the  objectives  in  the  best-possible  way. 
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3.3  Stage  3 

This  stage  will  develop  eLearning  material  for  eLearning  courses  and  need  answer  following  questions. 

• How  to  develop  added  value  (multimedia,  animation,  graphics,  interactive  audio,  video,  simulation 
units)  for  eLearning  content? 

• How  can  learning  progress  be  checked  in  an  eLearning  course? 

• How  can  you  give  learners  helpful  feedback  and  keep  a record  on  the  learning  objectives  individual 
learners  have  reached  at  any  given  time? 

• How  the  designs  of  technical  sheet  look  like? 


Stage  1 
Background 

Define  project  goal 

■ Why  do  we  need  E- 
Learning  ? 

■ Strategy  project 

■ LMS  project 

■ Course  project 

■ Define  financial  ability 

stage  2 

Develop  an  instructional 
design  concept 

■ Define  the  course  purpose 

■ Define  topic  and  course 

Stage  3 

Content  Development 

■ Prepare  course  content 
elements 

■ Preparation  of  Material 
for  use  on  the  Web 

■ Developing  Storyboard 

■ Checking  the  Learning 
progress 

objective 

■ Define  target  group 

■ Needs  Assessment 

■ Characteristics  of  e-Learning 

■ e-Leaming  Scenarios 

r 

i 

Stage  4 
Tutoring 

■ Train  the  tutors 

■ Tutor  skill 

■ Communication  tool 

■ Interactivity 

Stage  5 

Start  the  eLearning 
course 

■ Course  management 

■ Technological 
management 

Stage  6 
Evaluation 

■ Formative  evaluation 

■ Summative  evaluation 

■ Confirmative 
evaluation 

Figure  1.  A model  for  e-learning  development  and  implementation 


3.4  Stage  4 

• What  possible  interactions  can  be  integrated  in  eLearning  and  what  should  you  be  aware  of  while 
planning  interaction  in  an  eLearning  package? 

• Why  is  tutoring  an  important  tool  of  eLearning,  and  what  are  the  requirements  and  tasks  a tutor 
needs  to  fulfill? 

• What  are  the  tools  a tutor  can  use  to  fulfill  these  tasks?  What  are  their  specific  features,  and  how  can 
they  be  used? 
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• What  forms  of  feedback  are  there,  and  what  rules  do  tutors  need  to  bear  in  mind  when  writing 
feedback? 

• What  are  the  factors  defining  a culture,  and  what  are  distinguishing  features  of  cultures? 

The  idea  of  using  e-learning  systems  was  focused  around  the  ability  to  connect  with  external  and  distance 
education  students  and  provide  greater  access  and  flexibility  to  these  students.  However,  e-learning  has  now 
become  a core  component  of  the  education  experience  for  many  students  in  higher  education  and  an  ever- 
increasing  combination  of  face-to-face  (F2F)  learning  and  e-learning  is  now  occurring.  This  learning, 
referred  to  as  blended  learning,  uses  technology  to  expand  the  physical  boundaries  of  the  classroom, 
providing  access  to  learning  content  and  resources  and  enhancing  the  instructor’s  ability  to  receive  feedback 
on  learners’  progress. 

The  most  significant  factor  in  the  study  was  that  of  structure  and  user  experience.  It  highlights  the  need  to 
design  computer  supported  collaboration  tools  that  encourage  student  interaction  to  produce  collaborative 
knowledge  building  through  communities  of  practice. 

The  new  generation  of  LMSs  will  not  focus  only  on  learning  content  creation,  delivery  and  assessment, 
but  will  try  to  include  collaborative  learning  and  teaching  methods.  This  new  approach  guarantees  the  rise  of 
students’  motivation  for  learning  and  leads  to  the  better  results.  All  learners  taking  an  LMS-based  course, 
regard-less  of  their  knowledge,  goals,  and  interests,  receive  access  to  the  same  educational  material  and  the 
same  set  of  tools,  buffered  with  no  personalized  support.  Students  passed  the  exam  from  the  first  try  with 
quite  high  average  mark. 


4.  CONCLUSION 

It's  indisputable  that  the  e-learning  is  the  most  efficient  and  sufficient  learning  method  based  on  technology. 

Now  our  immediate  problem  is  how  to  get  it  accustomed  correctly  to  our  learning  environment. 

E-learning  framework  we  present  you  is  based  on  the  most  efficient  and  expedient  e-learning  practice 
organized  worldwide.  German  organization  "Invest"  is  conducting  learning  in  this  direction  uninterruptedly 
since  2005  and  has  prepared  many  tens  of  specialists. 

We  carried  out  a study  of  700  students  from  University  of  Science  and  Technology  and  intended  to  show 
that  our  learners  are  disposed  or  not  for  e-learning.  And  now  it's  important  to  study  our  purposed  group 
before  introducing  e-learning  to  expose  which  learning  method  is  suitable  for  present  education  organization. 

Collaborative  online  learning  is  successful,  because  it  requires  continuous  active  participation  during  the 
academic  year  and  more  personal  responsibility  and  concentration  when  learning.  In  that  way,  this  approach 
to  learning  reduces  the  time  needed  for  preparing  the  exam,  contributes  to  successful  passing  of  the  exam  and 
ensures  deep  level  learning. 
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ABSTRACT 

When  a student  is  faced  with  uncertainty  in  the  trustworthiness  of  a learning  activity  to  meet  their  intended  learning 
goals,  it  may  cause  a student  to  have  a state  of  anxiety  and  a lack  of  confidence  in  the  teaching  activity.  A student’s  trust 
in  the  teaching  agents’  ability  to  provide  an  appropriate  teaching  activity  is  needed  to  reduce  the  student’s  uncertainty.  In 
this  paper,  we  present  a conceptual  trust  model  for  an  e-leaming  transaction.  The  trust  model  quantifies  how  much  of  the 
student’s  trust  should  be  given  to  a teaching  activity  by  using  a learning  outcome-based  trust  and  a reputation-based  trust 
technique.  A possible  utility  of  a teaching  activity  is  estimated  in  order  to  calculate  the  degree  of  a student’s  trust  in  an 
outcome-based  trust  technique.  It  is  hoped  that  the  trust  model  presented  in  this  paper  could  help  a student  choose  the 
trustworthy  material  without  any  recommendation  from  an  expert. 


KEYWORDS 

E-learning,  tmst,  tmst  in  e-learning 


1.  INTRODUCTION 

Uncertainty  presents  a difficulty  for  students  when  choosing  appropriate  teaching  activities  and  materials, 
especially  for  their  self-study.  A student’s  uncertainty  can  be  resolved  by  building  their  trust  in  the  teaching 
activities  and  materials  in  e-learning.  For  example,  trust  can  be  determined  to  choose  appropriate  learning 
resources  from  different  providers  in  recommender  systems,  or  used  to  explore  people’s  experience  to  find 
out  suitable  groups  of  collaborative  learning.  However,  the  research  of  trust  in  e-learning  is  still  lacking  in 
terms  of  the  evaluation  method  of  trust  in  e-learning/teaching  activities  (Yingchun;  Youli  2010).  This  paper 
is  aimed  at  proposing  a new  conceptual  trust  model  that  can  be  used  to  calculate  the  student’s  trust,  which  is 
the  degree  of  trust  in  a teaching  activity  within  an  e-learning  transaction,  from  the  perspective  of  the  student. 
The  model  aims  to  support  a student’s  decision-making  when  considering  trust  in  teaching  activities  and 
materials.  The  paper  is  organised  as  follows.  The  related  work  for  our  trust  model  is  given  in  Section  2, 
followed  in  Section  3 by  the  proposal  of  a conceptual  student’s  trust  model.  Finally,  conclusions  are  drawn  in 
Section  4. 


2.  RELATED  WORK 

An  e-learning  transaction  is  defined  as  the  consideration  of  a situation  in  which  a teacher  and  a student  are 
attempting  to  achieve  the  same  outcome  in  a specific  context  (Gilbert  & Gale,  2008).  An  e-learning 
transaction  model  is  composed  of  three  units:  an  intended  learning  outcome  (ILO),  a teaching  activity  and  a 
learning  activity.  First,  an  ILO  is  a statement  of  what  the  student  is  expected  to  know  and  will  be  able  to  do  at 
the  end  of  an  applied  learning  process.  An  ILO  must  be  clear  in  order  to  support  the  delivery  and  assessment 
planning,  such  that  students  can  recognise  what  is  happening  and  understand  how  to  cope  with  their  learning 
and  assessment  (Macdonald,  1999).  The  general  statement  of  an  ILO  is  presented  as,  “By  the  end  of  the 
course,  the  student  will  be  able  to  X,  where  A is  a performance”  (Gilbert  and  Gale,  2008).  For  example,  in  an 
e-learning  lesson,  an  ILO  is  defined  as  “the  student  will  be  able  to  analyse  target  audienee  eharacteristies  by 
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listing  those  characteristics  pertinent  to  the  e-leaming  under  consideration”.  The  performance  is  defined  as 
“...to  analyse  target  audience  characteristics  by  listing  ...”.  The  intended  learning  outcome  consists  of  two 
primary  elements:  subject  matter  and  capability.  Second,  a teaching  activity  is  defined  as  an  intentional 
activity  of  a teacher  whose  aim  is  to  bring  a learning  activity  (Hirst,  1971).  The  design  of  a teaching  activity 
is  considered  in  two  parts:  pedagogical  content  knowledge  and  types  of  teaching  activity.  The  pedagogical 
content  knowledge  in  teaching  is  sourced  from  teachers’  own  subject-matter  knowledge  of  taught  materials 
(Even  & Tirosh,  1995).  Several  kinds  of  teaching  material  are  used  to  represent  the  pedagogical  content 
knowledge  or  the  subject  matter,  such  as  an  assignment  in  a book,  an  example  in  a video  or  a definition  on  a 
website.  There  are  four  types  of  teaching  activity:  “showing”,  “telling”,  “asking”  and  “feeding  back”.  The 
teacher  needs  to  decide  which  teaching  activity  is  appropriate  for  the  student  based  on  the  student’s  prior 
competence  and  the  ILO.  Third,  a learning  activity  is  an  activity  performed  by  the  student  that  aims  to  bring 
about  the  achievement  of  a new  competence  by  using  an  existing  one  (Hirst,  1971).  In  other  words,  every 
student  learns  in  a unique  manner  and  the  new  competence  is  acquired  from  a prior  competence,  such  as  a 
cognitive  ability. 


3.  TRUST  BASED  ON  LEARNING  OUTCOME  AND  REPUTATION  (TOR) 

In  a learning  outcome-based  trust  technique,  the  antecedents  of  trust  are  derived  based  on  the  schema  of 
intended  learning  outcome  (sILO).  In  details,  the  sILO  is  a direct  acyclic  graph  of  ILOs 
(Tangworakitthaworn  et  ak,  2013).  The  sILO  is  given  by  an  expert  to  connect  tasks  between  the  teaching  and 
learning  activity.  If  we  know  exactly  how  an  expert  achieves  ILO  by  their  own  knowledge,  it  is  then  possible 
to  estimate  any  teaching  activity’s  trustworthiness  based  on  the  expert’s  knowledge.  When  the  schema  of 
intended  learning  outcomes  represents  the  expert’s  knowledge,  the  trustworthy  teaching  activity  will  be 
determined  from  the  sILO.  It  is  possible  for  students  to  estimate  a degree  of  trust  using  their  own  assumption 
and  trustworthiness  information.  The  proposed  conceptual  student’s  trust  model  is  aimed  at  helping  a 
teaching  agent  provide  the  evidence  of  trustworthiness  in  teaching  activities  and  materials  for  students.  The 
construction  of  a student’s  eoneeptual  trust  model  using  a learning  outcome-based  trust  and  reputation-based 
trust  technique  (TOR)  is  shown  in  Ligure  1.  The  antecedents  of  the  student’s  trust  model  are  grouped  into 
three  categories:  a student’s  assumption  to  trust,  a student’s  trust  perception  of  a teaching  activity,  and  a 
pedagogy  context  depending  on  the  student’s  expectation  of  ILO  (elLO).  The  details  of  each  antecedent  of  a 
student’s  trust  are  deseribed  below. 

3.1  Student’s  Assumption  to  Trust 

In  this  work,  ^student ’s  assumptions  to  trust’  is  defined  as  a notion  of  trust  that  a student  considers  to  be  true 
by  themselves.  Students  have  to  choose  an  appropriate  teaching  activity  in  order  to  acquire  or  improve  their 
technical  skills  or  new  subject  knowledge.  They  need  to  develop  their  own  opinions  and  make  a decision  to 
trust  any  new  information  themselves.  Therefore,  students  need  some  notion  of  trust,  so  that  they  can 
estimate  the  trustworthiness  of  a teaching  activity.  The  good  assumption  of  trust  arises  from  the  good  trust 
propensity  and  their  prior  competences  (Bigley  & Pearce;  1998  Mayer  et  ak,  1995;  Rotter,  1980). 

Mooradian  et  ak  (2006)  reported  that  individuals  with  a low  trust  propensity  perceive  others  as  self- 
centred,  conniving  and  potentially  dangerous.  People  who  have  a high  trust  propensity  believe  that  most 
people  are  sincere,  fair  and  have  good  intentions.  Trust  propensity  is  an  important  factor  that  influences  a 
trust  antecedent  in  new,  unusual,  uncertain  or  unstructured  situations  and  it  is  a measure  of  how  willing  an 
individual  will  be  to  trust  someone  before  they  have  any  information  about  that  person.  On  the  other  hand, 
most  of  the  time,  the  prior  competences  of  a student  are  not  enough  to  engage  the  new  learning  purpose  or  to 
use  it  to  bridge  new  learning.  Students  may  face  difficulty  in  decision-making  to  trust  in  a teaching  activity 
that  will  hopefully  engage  new  knowledge.  More  potential  of  a student’s  related  prior  competences  (pILO)  to 
the  teaching  activity  means  an  increase  in  student’s  trust  in  the  teaching  activity.  Therefore,  an  analysis  is 
needed  of  how  many  of  a student’s  prior  competences  are  potentially  relevant  for  acquiring  new  knowledge. 
The  teaching  agent  in  e-learning  should  recognise  the  knowledge  schema  to  understand  students’  background 
knowledge,  and  use  the  knowledge  schema  to  bridge  between  student’s  prior  competence  and  new  learning. 
Therefore,  the  student’s  trust  propensity  and  the  student’s  prior  competence  are  needed,  especially  when  the 
trust  prediction  has  little  or  no  information  about  a teaching  activity. 
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3.2  Trustworthiness  of  Teaching  Activity 


Past  research  has  suggested  that  trust  from  a student’s  perspective  is  their  willingness  to  believe  and  have 
confidence  in  a teaching  agent,  such  that  it  will  take  appropriate  steps  to  help  them  achieve  their  learning 
purpose  or  the  intended  learning  outcome  (Corrigan  and  Chapman,  2008).  The  teaching  activity  must  have 
the  following  characteristics  in  order  to  give  evidence  about  the  value  of  the  learning  experience: 


Student’s  Prior  Competencies 
(pILO) 


Student’s  Trust  Propensity 


Benevolence: 

Weighting  the  Relevance  between  Prior 
Competencies  (pILO)  and  Prerequisite  ILOs  of 
Teaching  Activity 

Competence: 

Weighting  the  Relevance  between  Student’s 
Expectation  of  ILO  (elLO)  and  ILO  of  Teaching 
Activity 

Integrity: 

How  Much  does  the  Completeness  of  Subject 
Matter  in  Teaching  Activity  compare  with  the 
Intended  Learning  Outcomes  Schema  (sILO). 


Reputation: 

Students’  Satisfaction  Rating 
For  What?: 

Identifying  Student’s  Expectation  of  ILO 

(elLO) 

For  Whom?: 

Identifying  Student’s  Prior  Competences 
(pILO) 

For  Where?: 

Distinguishing  where  Student  is  currently  at, 
Compared  with  Expert's  Knowledge  (sILO) 

For  When?: 

Identifying  Readiness  of  Student  to  Accept 
Challenge  in  Expert's  Knowledge  (sILO) 


Student’s  Assumption  to  Trust 


Student's  Perception  of  Teaching 
Activity’s  Trustworthiness 


Pedagogy  Context 


4 


< 

* 

Student’s  Trust 

< 

Type  of 
Composed  of 


Figure  1.  Conceptual  model  of  a student’s  trust  based  on  learning  outcome  and  reputation  (TOR  model) 

• Benevolence  is  the  belief  that  the  teaching  agent  cares  about  the  well-being  of  the  student.  A teaching 
activity  is  benevolent  if  it  is  provided  based  on  the  student’s  expectations  and  prior  competence.  The 
benevolent  characteristic  of  a teaching  activity  is  measured  by  weighing  the  relevance  between  the 
student’s  prior  competences  (pILO)  and  the  set  of  prerequisite  ILOs  of  each  teaching  activity  within 
the  schema  of  intended  learning  outcomes  (sILO). 

• Competence  is  a student’s  perception  that  the  teaching  agent  has  the  professional  knowledge  and 
abilities  to  fulfil  its  required  tasks.  Being  competent  could  be  measured  by  weighting  the  relevance 
between  the  student’s  expectation  of  ILO  (elLO),  and  the  ILO  of  each  teaching  activity  within  the 
schema  of  intended  learning  outcomes  (sILO). 

• Integrity  is  a student’s  perception  that  the  teaching  agent  shows  honesty  with  its  sources  of  material 
and  is  honest  to  the  student.  The  integrity  could  be  measured  as  the  completeness  of  subject  matter 
content  in  a teaching  activity  compared  with  the  schema  of  intended  learning  outcomes  (sILO)  and 
also  detail  of  the  source  material  used  in  the  teaching  activity,  for  example  author  details  and  material 
type,  location,  and  modification  date. 

• Reputation  is  how  much  respect  or  admiration  a teaching  agent  receives  for  giving  the  teaching 
activities,  based  on  past  transactions  and  interaction.  Being  reputable  could  be  measured  by  student 
satisfaction  ratings  that  focus  on  the  importance  of  the  teaching  agent  fulfilling  its  students’ 
expectations. 
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3.3  Structure  of  Pedagogy  Context 

Literature  has  shown  that  trust  formation  can  be  improved  by  using  context  data  (Jpsang  2005,  Trbovich  & 
Patrick  2004).  We  enhance  the  student’s  trust  model  by  considering  the  pedagogy  context  to  enrich  trust 
estimation  functions  with  the  possibility  of  analysing  contextual  information.  This  research  defines  the 
pedagogy  context  as  information  that  can  be  used  to  characterise  the  situation  of  a person  or  an  object  in  a 
pedagogical  environment.  The  pedagogy  context  is  constructed  by  the  teaching  agent  based  on  the  student’s 
expectation  of  ILO  (elLO)  and  the  expert’s  knowledge  represented  by  the  schema  of  intended  learning 
outcomes  (sILO).  The  pedagogy  context  is  categorised  into  four  parts: 

• What?  - What  a student  expects  or  wants  to  achieve.  This  is  represented  by  the  student’s  expectation 
of  ILO  (elLO)  including  the  surrounding  information  and  their  condition. 

• Whom?  - Distinguishing  the  characteristics  of  a student  by  knowing  how  much  knowledge  a student 
already  has  (pILO)  and  how  much  knowledge  a student  needs  in  order  to  capture  the  necessary  ILO  in 
the  schema  of  intended  learning  outcomes  (sILO).  Recognising  what  the  trustworthy  teaching  activity 
is  relevant  to  the  student’s  expectation. 

• Where?  - Distinguishing  where  a student  is  currently  at  in  the  schema  of  intended  learning  outcomes 
(sILO)  during  the  learning  processes.  Recognising  what  the  trustworthy  teaching  activity  is  relevant  to 
the  potential  of  where  that  student  could  be. 

• When?  - Identify  when  a student  is  ready  to  learn  each  teaching  activity.  Identifying  the  readiness  of  a 
student  to  accept  the  challenge  of  the  teaching  activities.  Recognising  when  the  teaching  activity  could 
be  delivered  to  the  student  based  on  the  student’s  expectation  of  ILO  (elLO),  the  student’s  prior 
competences  (pILO)  and  the  prerequisite  ILO  of  teaching  material. 


4.  CONCLUSION 

In  this  paper,  we  present  a conceptual  student’s  trust  model  that  shows  the  antecedents  of  a student’s  trust 
using  learning  outcome-based  trust  and  a reputation-based  trust  technique.  A student’s  trust  is  defined  as  the 
degree  of  trust  from  the  student’s  perspective,  and  it  shows  the  student’s  willingness  to  believe  and  have 
confidence  in  the  teaching  agent,  such  that  the  student  will  take  appropriate  steps  to  help  them  achieve  their 
intended  learning  outcome  (ILO).  In  more  detail,  the  stating  of  learning  purpose,  the  perception  of  evidence 
of  trustworthiness  and  the  student’s  own  assumptions  are  introdueed  as  antecedents  to  assign  a student’s  trust 
degree  in  a teaching  activity.  The  proposed  trust  model  is  aimed  at  estimating  the  student’s  trust  degree  in 
teaching  activity  even  if  the  student  has  no  opinion  and  without  any  recommendation  from  an  expert.  In 
future  work,  the  TOR  model  will  employ  the  fuzzy  logic  to  calculate  the  degree  of  the  student’s  trust,  and  be 
validated  via  a user  evaluation.  The  accuracy  and  precision  of  the  TOR  model  will  also  be  quantified. 
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ABSTRACT 

Analysis  of  learning  outcomes  can  be  a complex  and  esoteric  instructional  design  process  that  is  often  ignored  by 
educators  and  e-learning  designers.  This  paper  describes  a model  of  analysis  that  fosters  the  real-life  application  of 
learning  outcomes  and  explains  why  the  model  may  be  needed.  The  Elemental  Learning  taxonomy  is  a hierarchical 
model  composed  of  two  primary  types  of  learning:  (a)  elemental  learning  composed  of  actual  and  simulated  elements; 
and  (b)  synthetic  learning,  composed  of  procedural  understanding,  conceptual  understanding,  and  related  knowledge.  The 
importance  of  iteration  and  intuition  in  learning  analysis  is  also  discussed. 
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1.  INTRODUCTION 

A primary  assumption  of  meaningful  e-learning  is  the  fidelity  of  design  to  learning  outcomes.  The  focus  of 
this  paper  is  that  the  primary  design  of  assessment  and  learning  activities  that  match  learning  outcomes 
should  emphasize  elemental  (real-life  or  simulating  real-life)  learning.  The  emphasis  on  real-life  learning 
outcomes  is  proposed  from  the  perspective  of  a simpler  taxonomical  model  that  with  five  levels  that  can  be 
easily  grasped  by  non-instructional  designers  and  applied  to  the  analysis  of  meaningful  learning  outcomes. 

More  than  ever  it  is  possible  to  define  learning  progress  along  a “trajectory  of  experience”  (Greeno, 
1997).  For  more  than  a century,  a conceptual  approach  related  to  guiding  the  learning  experience  has  been 
argued  along  a loose  continuum.  One  approach,  beginning  perhaps  with  Thorndike’s  specific  transfer  theory 
of  identical  elements  (Thorndike  & Woodworth,  1901),  has  emphasized  situated  learning  outcomes.  This 
approach  emphasizes  task-based  learning  and  probably  because  of  its  early  association  with  logical 
positivism  and  ‘objective’  use  of  data  has  been  overly  vilified  by  the  overgeneralization  of  early  behaviorists. 
The  other  end  of  the  continuum  supports  the  transfer  of  general  skills  and  principles  to  promote  learning. 
This  was  characterized  early  on,  for  example,  by  Gestalt  psychologists  such  as  Wertheimer  (1945)  and 
Katona  (1940).  Because  of  the  decontextualized  nature  of  the  general  skills  and  principles  approach, 
inductive  learning  philosophies  (e.g.,  constructivism  and  connectivism)  the  common  sense  inclination  of 
many  educators  is  to  see  the  merit  in  both  positions  (e.g.,  by  De  Corte,  1999).  Few  would  argue,  however, 
that  learning  activities  and  assessments  that  provide  a situated  context  are  more  likely  to  retained  in  long- 
term memory  and  more  able  to  be  applied  in  a real-life  environment.  Although  we  keep  reinventing  it,  the 
theory  and  research  supporting  these  ideas  have  been  very  well-defined  as  early  as  the  days  of  Thorndike  and 
Dewey.  The  challenge  is  to  apply  these  ideas  in  a simpler  approach  for  analyzing  teaching  and  learning 
outcomes  in  such  a complex,  information-bloated  environment  as  e-learning. 
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2.  LEARNING  ANALYSIS  AND  LEARNING  TAXONOMIES 

Learning  analyses  are  formal  approaches  to  clarify  the  rational  relationships  among  learning  outcomes.  In 
terms  of  intentional  learning  environments,  such  as  higher  education  or  technical  training,  two  methods  have 
been  employed  with  some  effectiveness:  taxonomies  and  rule-based  frames. 

Using  a learning  taxonomy  is  appropriate  when  content  is  characterized  by  rational  relationships.  Classic 
learning  taxonomies  include  those  by  Bloom  (1956)  and  Gagne’  (1985).  The  approach  here  is  to  ask  in  effect, 
“If  you  were  to  assess  what  the  learner  knows  or  can  do,  how  would  know?”  The  taxonomies  of  Bloom  and 
Gagne’  provide  an  rational  way  to  approach  that  question  by  defining  learning  outcomes  as  abstract  concepts 
and  identifying  the  nature  of  the  intentional  or  incidental  (or  unintended)  learning  outcomes  that  provide  the 
learning  as  well  as  the  relationships  among  the  learning  outcomes.  Figure  1 illustrates  Gagne’s  intellectual 
skills  subdomain. 


Problem-Solving 


1 

Rules 

Rules 



Concepts 


Concepts 


Concepts 


• Discriminations 

Discriminations 

Figure  1.  Gagne’s  Intellectual  Skills  Subdomain 

As  you  can  see,  in  this  taxonomy  problem-solving  or  higher-order  skills  depend  on  rule  or  procedural 
prerequisites;  rules  depend  on  concepts,  both  abstract  (e.g.,  love  or  identity)  and  concrete  (e.g.,  table  or  dog); 
and  concepts  require  discriminations.  This  approach  is  hierarchical  and  deductive.  It  has  been  used  routinely 
by  skilled  instructional  designers  to  analyze  what  must  be  learned  or  assessed.  It  is  very  effective  in 
indicating  the  essential  prerequisites  for  a course  or  e-learning  module  (Gagne  and  White,  1978) 

Another  approach  to  learning  analysis  is  a rule-based  frame.  A rule -based  frame  is  a superb  way  to 
structure  intelligent  computer-based  learning  scenarios  or  examples  and  has  increasingly  been  employed  in 
computer-adaptive  examinations.  Figure  2 illustrates  a matrix  of  related  content  that  has  been  used  for 
structuring  question-based  or  declarative  examples,  case  studies,  identifying  misconceptions,  and  so  forth. 
When  designed  in  rationally  appropriate  algorithms  for  computer  software,  rule-based  frames  (sometimes 
referred  to  as  Type  2 Frames)  identify  specific  classification  problems,  such  as  misconceptions  or 
overgeneralizations)  and  adapt  by  providing  classification  examples  that  emphasize  appropriate  levels  of 
discrimination  (telling  the  difference  between  rationally  related  content)  and  generalization  (applying  to 
increasingly  complex  or  divergent  examples). 
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Figure  2.  An  example  of  a rule-based  frame  (from  Dempsey  & Driscoll,  1996) 

The  advantages  of  using  classic  learning  taxonomies  or  rule-based  frames  to  analyze  frames  are  several. 
They  provide  a systematic  approach  to  thinking  about  learning  content  for  assessment  or  structuring  rational 
relationships  including  essential  prerequisite  knowledge  and  skills.  They  can  be  empirically  verified  by 
learner  responses.  In  addition  they  have  relevance  for  advanced  e-learning  software  applications  including 
artificial  intelligence,  computer-based  testing,  computer-based  testing,  and  systems-type  wizards. 
Unfortunately,  these  approaches  are  not  used  at  all  or  used  ineffectively  by  most  educators.  Simply  put,  they 
are  just  too  complicated  to  be  easily  applied  by  content  experts  and  even  less-skilled  instructional  designers. 


3.  ELEMENTAL  AND  SYNTHETIC  LEARNING 

Another  approach  to  learning  analysis  and  design  is  to  emphasize  elemental  (real-life  or  simulating  real-life) 
outcomes  and,  when  these  are  identified,  look  for  those  synthetic  outcomes  that  support  their  learning. 
Elemental  Learning  refers  to  the  actual  or  real  tasks  applied  in  a real  or  close  proximate  environment  in 
which  the  learning  outcomes  will  be  used.  They  are  context-  and  content-specific.  They  can  also  contribute  to 
learning  similar  elemental  outcomes  by  virtue  of  the  learner’s  enhanced  experiential  schema.  An  obvious 
example  for  the  need  for  elemental  learning  involves  the  training  of  airline  pilots.  Who  would  want  to  be  a 
passenger  in  an  airplane  in  which  the  pilot  had  not  experienced  learning  in  an  actual  passenger  airliner? 

By  contrast,  synthetic  learning  outcomes  are  the  cognitive  learning  outcomes  necessary  to  support 
elemental  learning.  Synthetic  learning  outcomes  are  less  context-specific,  and  the  learner’s  experience  is 
often  less  important  in  acquiring  these  outcomes.  Synthetic  here  refers  to  forming  something  new  (elemental) 
by  combining  other,  usually  decontextualized,  outcomes.  These  are  the  traditional  learning  outcomes.  In 
taxonomies  such  as  Bloom’s  (1956)  or  Gagne’s  (1985),  these  are  traditionally  believed  to  be  hierarchical,  i.e., 
learning  rules  requires  learning  concepts,  some  basic  knowledge  is  required,  and  so  forth. 

Figure  3 illustrates  a simpler,  more  direct  taxonomy  that  focuses  on  elemental  learning  outcomes.  After 
elemental  learning  outcomes  are  identified,  synthetic  learning  outcomes  that  support  them  are  added. 
Elemental  learning  provides  the  context.  Synthetic  learning  outcomes  support  the  context. 
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Figure  1.  Elemental  learning  taxonomy  represented  as  a pyramid  culminating  in  actual  elements.  The  top  two  levels  are 
elemental  learning  outcomes.  The  bottom  three  learning  outcomes  are  viewed  as  synthetic 


3.1  Components  of  the  Elemental  Learning  Taxonomy 

Actual  Elements  are  the  most  direct  measures  of  the  real  learning  outcomes  required  by  the  real  environments 
in  which  the  learner  needs  to  use  those  learning  outcomes. 

Simulated  Elements  promote  learning  transfer.  They  reproduce  reality  or  some  version  of  reality  “close 
enough”  to  actual  elements.  Although  simulated  elements  may  be  lacking  in  some  essential  characteristics  of 
reality,  they  retain  the  context  of  reality  to  some  purposeful  extent. 

Procedural  understanding  requires  that  learners  “walk  the  walk,”  not  just  “talk  the  talk.”  Even  though 
procedural  learning  can  be  very  complex  it  is  missing  essential  features  of  the  context.  Even  so,  it  is  a “DO” 
learning  outcome  versus  a “Know”  outcome. 

Conceptual  Understanding  can  be  abstract  or  physical.  It  can  be  learned  (and  assessed)  by  novel  examples 
and  nonexamples,  by  metaphors,  and  by  inductive  observation  and  reflection. 

Related  Knowledge  associated  information  is  intentional  (e.g.,  from  an  Internet  resource  accessed  via  a 
handheld  mobile  device)  or  incidental  (e.g.,  from  a television  program  on  another  subject). 

These  categories  are  not  exhaustive.  Eor  example,  motor  skills  and  affective  learning  are  certainly 
important.  Motor  skills  and  affective  learning  are  also  contextual  and  almost  always  are  highly  associated 
with  elemental  learning.  Eor  example,  how  do  we  learn  an  attitude?  One  way  we  do  is  to  intentionally  or 
unconsciously  emulate  a human  or  human-like  entity  in  our  life  (a  mentor,  perhaps)  or  in  a simulated 
experience  (e.g.,  an  actor  in  a movie).  It  is  the  real  or  simulated  experience  that  guides  the  choices  we  make 
that  exhibit  the  attitude  learning  that  has  taken  place. 


3.2  Iteration  and  Intuition 


Elemental  learning  has  a theoretical  foundation  in  the  notion  of  spiral  curriculum  popularized  by  Jerome 
Bruner  (1960).  In  a spiral  curriculum,  the  learner  revisits  real  or  simulated  learning  situations  iteratively  and 
each  time  the  learning  deepens  by  building  on  prior  experience  (p.l3).  Complex  learning  outcomes  (e.g., 
negotiating  effectively  in  a foreign  language  in  an  alien  culture)  require  the  gradual  scaffolding  offered  by 
iterative  elemental  learning  scenarios.  Experiential  iterations  of  these  sorts  also  subsume  and  give  meaning  to 
supportive  synthetic  learning  outcomes.  In  a spiral  curriculum  challenge  (difficulty)  is  increased  as  the 
learner’s  competence  increases  through  iteration. 

Iterative  processes  of  elemental  learning  also  promote  “intuitive  leaps.”  Bruner  often  referred  to  experts 
performing  real  tasks  where  they  “leap  intuitively  into  a decision  or  to  a solution  to  a problem”  (1960,  p.62). 
In  medicine  and  other  areas,  such  as  physics,  intuitive  leaps  are  referred  to  as  the  forward  reasoning  (from 
data  to  solution)  applied  by  experts  [7]  versus  the  backwards  (rigidly  algorithmic)  reasoning  of  novices. 
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4.  CONCLUSION 

The  elemental  learning  model  does  not  exist  in  a vacuum.  Its  goal  is  to  provide  a simpler  method  of  analysis 
that  result  in  significant  learning  outcomes.  Established  learning  analysis  models  (e.g.,  Gagne’s  Taxonomy  or 
rule-based  framing)  can  be  employed  effectively  to  promote  specific  learning  outcomes.  Unfortunately,  these 
models  often  result  in  decontextualized  learning.  They  take  some  expertise  to  use  in  educative  practices  that 
result  in  increased  learning  retention.  Additionally,  they  can  be  cumbersome  to  apply  and  thus  the  important 
practice  of  learning  analysis  is  often  neglected. 

The  elemental  learning  framework  situates  learning  in  reality  (actual  elements)  or  a mockup  of  reality 
(simulated  elements),  and  does  so  iteratively  to  the  extent  that  is  possible.  In  analyzing  a course  or  a module 
or  some  other  e-learning  situation,  the  challenge  is  to  move  beyond  “giving”  decontextualized  knowledge 
and  general  principles  (synthetic  learning  outcomes).  The  elemental  learning  framework  provides  a structure 
for  contextual  learning  that  is  so  often  lacking  in  e-learning  environments.  Actual  and  simulated  learning 
outcomes  supported  by  synthetic  learning  outcomes  create  the  structure  without  over-controlling  the  natural 
challenges  of  meaningful  learning  experiences. 
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ABSTRACT 

The  number  of  visually  impaired  people  is  increasing  year  by  year.  Although  attention  has  been  given  to  the  needs  of 
people  with  disabilities,  most  of  the  discussion  has  focused  on  social  welfare,  while  talk  about  assistive  technology  for 
people  with  disabilities  is  rare.  The  blind  need  training  courses  for  reconstruction  and  rehabilitation.  Orientation  and 
mobility  (O&M)  training  is  important  for  visually  impaired  people,  teaching  safe,  efficient  and  effective  travel  skills. 
Skills  learned  from  O&M  training  courses  can  help  the  blind  walk  on  the  street  safely.  Crossing  the  street  is  especially 
dangerous,  since  blind  people  cannot  see  traffic  lights,  and  rely  mostly  on  sound  for  information  about  their  environment. 
Thus,  learning  to  recognize  the  varied  sounds  of  vehicles  and  determining  the  direction  and  speed  of  moving  vehicles  is 
critical.  In  this  paper,  we  propose  an  interactive  game  with  3D  sound  that  simulates  a busy  street  environment.  The 
proposed  game  tries  to  build  a virtual  environment  with  3D  sound  to  help  visually  impaired  people  learn  to  cross  the 
street  safely.  As  the  proposed  training  game  is  designed  for  the  blind,  the  technologies  of  Kinect  and  Text-to-Speech 
(TTS)  are  used  in  the  human-computer  interface  in  the  proposed  game,  so  that  they  can  use  the  game  independently. 
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1.  INTRODUCTION 

Vision  is  the  most  important  and  natural  way  humans  receive  information  from  the  environment.  We  rely  on 
vision  to  handle  most  of  our  tasks  every  day.  Those  with  adventitious  blindness  depended  on  vision  to 
receive  most  messages  from  their  environment  before  going  blind,  so  it  is  difficult  for  them  to  adapt  when 
they  lose  their  eyes.  For  example,  if  we  drop  a key,  we  will  look  for  the  key  with  our  eyes  and  pick  it  up. 
Visually  impaired  people,  on  the  other  hand,  have  to  listen  to  the  sound  when  a key  falls.  Today,  the  number 
of  visually  impaired  people  is  increasing  year  by  year.  Although  a great  deal  of  attention  has  been  given  to 
the  needs  of  people  with  disabilities,  most  of  the  discussion  has  been  about  social  welfare  rather  than 
assistive  technology  for  people  with  disabilities.  Training  courses  in  reconstruction  and  rehabilitation  allow 
the  blind  to  lead  a more  independent  life. 

Orientation  and  mobility  (O&M)  [1]  is  important  training  for  visually  impaired  people.  O&M  is  a 
profession  that  focuses  on  instructing  individuals  who  are  blind  or  visually  impaired  in  safe  and  effective 
travel  through  their  environment  [2].  O&M  training  research  is  carried  out  by  medical  and  special  education 
researchers,  however,  most  O&M  training  needs  extensive  space  and  expensive  facilities.  At  the  Institute  for 
the  Blind  of  Taiwan,  for  example  [3],  the  trainee  needs  a large  area  in  which  to  train.  The  instructor  assists 
the  blind  person  in  learning  to  use  a white  cane  or  guide  dog  to  walk  on  the  road  [4].  After  training,  the  blind 
person  can  sense  his  or  her  location  by  noise,  smell  and  sound  direction.  The  limitation  is  that  the  visually 
impaired  person  cannot  learn  alone,  an  instructor  must  assist  with  the  training.  Our  goal  is  to  develop  O&M 
training  for  blind  people  using  assistive  technology.  We  want  the  training  to  be  entertaining,  game-based,  and 
easy  for  visually  impaired  people  to  use  by  themselves. 
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In  this  paper,  we  report  the  development  of  an  Auditory  Perception  Re-establishment  Training  System 
(APRETS)  for  visually  impaired  people.  APRETS  can  simulate  the  traffic  on  a busy  street  environment, 
assisting  the  blind  to  learn  to  cross  the  street  using  3D  sound.  APRETS  is  simple  to  operate  by  motion 
detection,  using  the  Kinect  system.  The  rest  of  the  paper  is  organized  as  follows.  Previous  work  related  to 
APRETS  is  presented  in  Section  II.  The  state  of  the  art  in  software  in  discussed  in  Section  III.  This  is 
followed  by  the  implementation  of  APRETS  in  Section  IV.  We  had  visually  impaired  people  test  our  system, 
and  report  the  results  of  the  test  in  Section  V.  Einally,  the  conclusions  of  this  work  are  given  in  Section  VI. 


2.  BACKGROUND  AND  RELATED  WORKS 

APRETS  uses  3D  sound  technology  to  simulate  real  3D  traffic  sounds,  which  are  synthesized  using 
computerized  techniques  [5].  Most  3D  sound  techniques  are  based  on  head-related  transfer  function  (HRTE) 
[6].  HRTE  is  based  on  the  principles  of  Interaural  Time  Differences  (ITD),  Interaural  Intensity  Differences 
(IID)  and  the  Pinna  Effect  to  generate  3D  sound  [7].  In  reality,  if  you  hear  one  sound,  the  time  and  the 
intensity  of  the  sound  detected  is  slightly  different  in  the  two  ears,  allowing  us  to  judge  the  sound  direction. 
Software  allows  us  to  duplicate  this  effect  using  HRTE.  As  the  technology  improves,  the  sound  cards  already 
in  personal  computers  will  allow  us  to  hear  3D  sound. 

APRETS  uses  a motion  detection  to  operate  the  system.  The  motion  detection  system  we  used  is 
“Kinect,”  introduced  by  Microsoft  in  2010  [8,  9].  As  opposed  to  traditional  game  control  systems,  Kinect 
uses  body  motion,  allowing  the  player  to  control  the  game  instinctively.  Using  the  same  properties,  Kinect 
can  be  used  in  other  domains,  such  as  education  and  rehabilitation  [10,  11].  The  Kinect  sensor  is  a horizontal 
bar  connected  to  a small  base  with  a motorized  pivot  and  is  designed  to  be  positioned  lengthwise  above  or 
below  the  video  display.  The  device  has  an  RGB  camera,  depth  sensor  and  CMOS  running,  which  provides 
full-body  3D  motion  capture. 

In  the  Kinect  development  kit,  we  used  “OpenNI”  (Open  Natural  Interaction),  a multi-language,  cross- 
platform framework.  Currently  supported  platforms  are  Windows,  Mac,  and  Linux  [12].  OpenNI  has  a three- 
layer  concept.  The  first  layer  represents  the  interaction  applications.  The  second  layer  represents  the 
middleware  components.  The  third  represents  the  hardware  device  such  as  a microphone,  color  camera  and 
3D  depth  camera.  In  OpenNI,  there  are  two  types  of  Production  Nodes;  one  is  a Sensor  Related  Production 
Node,  the  other  one  is  the  Middleware  Related  Production  Node.  Therefore,  OpenNI  can  run  in  three  modes: 
(1)  full  body  analysis,  (2)  hand  point  analysis,  (3)  gesture  detection  and  (4)  scene  analyzer.  We  use  these 
modes  in  our  system. 

Text-to-speech  (TTS)  is  used  in  the  human-computer  interface  in  APRETS.  The  TTS  system  converts 
normal  language  text  into  speech  [13].  Through  many  years  of  development  of  TTS,  the  output  speech  is 
becoming  quite  fluent.  Different  TTS  engines  can  support  different  languages,  but  our  TTS  engine  outputs 
English  speech. 

APRETS  is  an  interactive  game  for  visually  impaired  people,  so  our  goal  is  “Game  Accessibility”  [14].  In 
general,  intuitive  operation  in  game  design  has  three  components  [15]:  (1)  Continuous  presentation,  (2) 
Physical  actions,  (3)  Reversible  actions  with  immediate  feedback.  Prom  the  above,  we  can  know  the  message 
present,  the  operating  mode  and  the  immediate  feedback  are  game  design  points.  Por  visually  impaired 
people,  our  goal  was  to  create  an  Audio  Game,  designed  with  the  following  capabilities  [16]: 

• Voice  Navigation 

• Voice  can  repeat 

• Easy  Control 

• Different  levels 

• Objects  Hint 

The  interactive  game  that  we  proposed  has  the  following  features: 

• No  special  space  is  needed  for  training;  APRETS  can  be  used  in  anywhere. 

• No  expensive  facilities  are  needed;  it  uses  only  headphones,  Kinect  and  a PC. 

• Easy  to  use;  visually  impaired  people  can  operate  it  independently  without  help. 

• 3D  Sound  technology  is  used  in  the  game;  3D  sound  simulates  the  environment  of  busy  street  traffic, 
with  many  different  vehicles  simulated. 

• Supports  speech  feedback. 

• Visually  impaired  people  can  use  Kinect  as  a special  human  interface. 
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Our  system  can  reduce  the  expense  of  O&M  training  for  blind  people  and  give  them  higher  motivation  to 
participate  in  the  training.  APRETS  uses  a text-to-speech  engine  so  blind  people  can  accept  and  control 
APRETS  easily. 
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Figure  1.  The  system  architecture. 
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Figure  2.  Hands  up  for  calibration. 


Figure  3.  The  game  situation. 


3.  SYSTEM  DESIGN 

We  developed  an  interactive  game  for  visually  impaired  people  to  use  Kinect  and  3D  Sound  for  O&M 
training.  To  make  the  game  more  interesting,  we  added  music  and  speech. 

3.1  Experimental  Environment 

1)  Motion  detection  - Kinect:  Nintendo’s  “Wii”  made  Motion  Sensing  Game  gaming  popular  [17,  18], 
but  Microsoft  Kinect  is  more  popular  worldwide.  Kinect  doesn’t  need  a controller,  as  your  body  is  the 
controller,  allowing  you  to  play  games  more  simply  and  more  intuitively.  We  like  this,  so  in  our  system  we 
used  Kinect  to  control  the  system. 

2)  Wireless  Headphone:  Our  system  requires  3D  Sound  and  freedom  of  movement,  so  wireless 
headphones  are  necessary. 

3)  Sound  Card:  The  sound  card  must  support  3D  sound.  The  Creative  sound  card  uses  CMSS-3D 
surround  sound  technology.  CMSS-3D  has  been  used  to  make  spatial  sound  [19],  so  we  chose  to  use  the 
Creative  sound  card  output  for  3D  sound. 

4)  Personal  Computer:  APRETS  can  run  on  any  computer  running  a Windows  platform.  Eor 
convenience,  we  used  a laptop  when  doing  the  test.  3.2  System  Architecture 
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In  this  paper,  the  computer  is  our  data  server.  APRETS  was  developed  on  a Windows  platform  with  four 
parts:  User  interface,  3D  sound  module.  Motion  detection  driver  and  APRETS  kernel  module.  The  system 
architecture  is  presented  in  Figure  1 . 

1)  User  interface:  Following  voice  hints,  the  user  makes  the  correct  pose  to  operate  the  system.  Since 
the  system  is  for  visually  impaired  people,  we  don’t  need  graphics  on  screen.  In  the  game,  we  have  sound 
hints  and  voice  hints.  The  user  can  put  his  hands  up  and  swing  his  arm  to  operate  the  system. 

2)  3D  Sound  module:  The  system  uses  FMOD  produced  3D  Sound.  FMOD  is  a proprietary  audio 
library  [20],  a popular  and  powerful  cross -platform  interactive  audio  system.  The  FMOD  software  was  used 
with  the  Creative  sound  card  hardware  [21]. 

3)  Motion  detection  driver:  The  motion  detection  system  receives  skeleton  information  and  position 
information.  This  information  is  interpreted  by  the  APRETS  kernel  module.  We  exploited  the  software 
development  kit,  OpenNI  to  develop  this  module. 

4)  APRETS  kernel  module:  The  APRETS  kernel  module  controls  data  transfer  and  training.  Feedback 
hints  from  the  user’s  motion  and  progression  feed  back  to  change  the  sound  source  and  position. 

3.2  The  Flowchart  of  the  Game 

We  designed  a game  like  the  classic  videogame  “Frog  crossing.”  The  traffic  is  different  from  “Frog 
crossing,”  in  that  the  sounds  of  bike,  motorcycle,  car  and  truck  are  produced  using  3D  sound,  and  there  are 
many  vehicles  in  one  game,  since  it  is  designed  to  help  visually  impaired  people  react  appropriately  in  real 
situations.  We  hope  to  help  blind  people  learn  to  recognize  hints  about  traffic  including  each  vehicle’s 
direction,  distance  and  speed  using  the  APRETS  system. 

The  game  flow  chart  is  simple,  as  we  don’t  want  to  make  it  too  hard  to  play  the  game.  The  game  begins 
with  calibration  of  the  Kinect  system  with  the  user  placing  his  or  her  hands  up  as  shown  in  Figure  2.  When 
calibration  is  completed,  the  system  will  explain  verbally  how  to  play  the  game. 

The  game  situation  is  presented  in  Figure  3.  There  are  three  roadways  in  the  game.  The  first  roadway  has 
only  bikes,  going  at  different  speeds  and  appearing  at  different  times.  This  is  the  easiest  roadway,  with  the 
bikes  looping  three  to  four  times.  The  user  can  cross  safely  after  the  bikes  pass.  In  this  roadway,  we  want  the 
user  to  learn  to  judge  the  bikes’  path  and  get  used  to  the  system.  Cars  and  motorcycles  appear  in  the  second 
roadway,  so  the  speeds  are  faster  and  it  is  harder  to  cross  than  the  first  roadway.  In  the  second  roadway,  we 
want  the  user  to  judge  both  the  speed  of  vehicles  and  type  of  vehicles.  The  hardest  level  is  the  third  roadway, 
which  has  more  cars  and  trucks,  at  faster  speeds  than  the  second  roadway.  Once  the  user  is  able  to  cross  the 
third  roadway,  he  or  she  has  successfully  completed  the  game. 

When  the  game  is  over,  the  system  will  calculate  the  time  spent  playing  the  game  and  tell  the  user.  If  you 
are  unfortunate  and  have  a traffic  accident  in  the  game,  there  will  be  a sound  of  brakes,  the  game  is  over,  and 
it  will  tell  you  how  much  time  you  spent  playing  the  game.  If  you  succeed,  you  can  hear  a short  section  of 
jazz  music  after  completing  the  game.  We  want  the  user  to  feel  exhilaration  in  playing  APRETS  as  they  learn 
how  to  navigate  through  traffic  on  the  roadway  as  part  of  O&M  training.  Since  people  love  games,  we  hope 
they  will  love  this  training. 


4.  CONCLUSION 

In  this  paper,  we  report  the  development  of  an  interactive  training  game  for  visually  impaired  people,  and  use 
of  that  interactive  game  for  O&M  training.  The  game  can  simulate  a street  environment  with  busy  traffic. 
APRETS  is  a computer-based  game  that  visually  impaired  people  can  play  with  Kinect  and  wireless 
headphones.  We  have  designed  a user-friendly  interface  which  uses  text-to-speech  and  voice  to  guide  the 
blind  user  in  using  APRETS.  After  the  game  prototype  was  completed,  we  invited  five  visually  impaired 
persons  to  test  the  proposed  game.  We  completed  a questionnaire  and  interview  with  the  subjects  after  the 
experiment.  Most  subjects  were  satisfied  with  the  training  game.  The  overall  satisfaction  was  3.88.  The  users 
also  gave  some  feedback  and  suggestions  for  system  improvement.  All  users  thought  the  game,  played  with 
Kinect  and  3D  sound,  was  really  novel  and  interesting.  In  the  future,  we  will  extend  the  functions  of  the 
training  game  for  other  O&M  training. 
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ABSTRACT 

The  method  of  competences  level  automated  management  is  developed  for  student's  competences  formation.  An 
insufficient  level  of  schools  graduates  in  physics  is  identified  as  the  modern  problem  of  the  education.  Teacher  does  not 
specify  which  step  in  a solution  a student  can  not  cope  with,  and  the  student  reiterates  his  mistake  in  other  tasks  solving. 
It  is  propose  to  prepare  students  to  a physics  examination  by  means  of  tasks  solution  competences  formation.  The  special 
software  for  intellectual  support  of  optimal  solution  of  analytical  physical  tasks  was  developed. 


KEYWORDS 

E-learning,  tasks  solution,  competence  level,  competences  formation. 


1.  INTRODUCTION 

According  to  Russian  government  program  “Development  Program  2020”one  of  the  main  objectives  is  the 
innovative  people-centered  development  of  science  and  technology  sectors.  For  the  solution  of  the  posed 
problem  in  education  the  priority  is  given  to  technical  and  natural  science  disciplines. 

The  implemented  Unified  State  Exam  (the  USE)  has  its  goal  to  select  capable  school  and  university 
graduates.  According  to  the  results  of  the  USE  2012  in  Physics  it  has  been  found  out  that  26,6%  (Ershov 
A.G.  2012)  of  all  the  school  graduates  got  average  and  high  grades  and  the  rest  of  the  school  graduates 
(73,4%)  got  minimal  and  low  grades  (Figure  1). 

The  percentage  of  graduates  in  the  score  USE  on  the 
physics  exam 

The  minimum 
and  the  lo\wst 

High  and  ^ 

medium; 


Figure  1.  Results  of  the  USE  2012  in  Physics 

There  has  been  observed  a tendeney  of  inereasing  number  of  sehool  graduates  who  have  not  passed  the 
minimum  threshold  (Figure  2a).  The  situation  is  no  better  with  those  who  got  100  grades  (Figure  2b).  As  a 
result  the  universities  specializing  in  engineering  technology  missed  out  on  competitive  applicants  in  2012. 

In  accordance  with  the  requirements  of  the  Federal  State  Edueation  Standard  all  school  graduates  must  be 
able  to  apply  their  knowledge  and  skills  in  familiar,  ehanged  and  new  situations.  The  formation  of  the  ability 
to  apply  knowledge  and  skills  in  a familiar  situation  can  be  achieved  by  doing  similar  tasks.  Application  of 
knowledge  and  skills  in  changed  and  new  situations  requires  some  extra  abilities. 

Teachers  of  Physics  tend  to  prepare  students  for  the  examination  by  giving  them  a great  number  of 
challenge  tasks.  Doing  such  tasks  students  acquire  knowledge,  the  very  knowledge  that  they  apply  at  the 
examination.  This  preparation  method  is  considered  ineffective  because  one  needs  much  time  for  fulfdling 
the  tasks  and  nobody  can  guarantee  that  students  will  be  able  to  cope  with  the  task  at  the  exam.  That  is  the 
reason  why  they  suggest  students  being  prepared  through  the  formation  of  task-doing  competences. 


351 


ISBN:  978-972-8939-88-5  © 2013  lADIS 


Figure  2.  a)  Failed  results  of  the  USE  2012  in  Physics;  b)  Maximum  results  of  the  USE  2012  in  Physics 

Due  to  the  development  of  new  technologies  and  new  PC  hardware  in  educational  institutions  the 
preparatory  work  should  be  done  through  automation  of  a teaching  process. 

Researchers  (Cecilia  Rossignoli,  Maria  Ferrara  and  Luisa  Varriale,  2013,  2013;  Margot  McNeill,  Maree 
Gosper  and  John  Fledberg,  2010)  propose  methods  of  material  assessment,  strategies  and  technologies  that 
enable  the  learner  to  embark  on  a learning  process. 

E-leaming  platforms  are  now  widely  used  by  educators  to  enhance  the  learners’  interest  in  learning, 
shorten  the  learning  portfolio,  and  improve  the  overall  learning  outcome  (Silvia  Knittl  & Hans  Pongratz, 
2010;  Said  Hadjerrouit,  2013).  Individual  learning  paths  may  be  used  and  suggested  to  user  in  dependence  on 
a given  answer  of  an  exercise  (Julia  Schrock  et  al,  2010;  Stefanie  Sieber  & Andreas  Henric,  2010;  Najat 
Smeda,  Eva  Dakich  andNalin  Sharda,  2010;  Vangel  V.  Ajanovski,  2013). 

However,  the  question  of  the  organization  the  e-leaming  practice-oriented  training  remains  unresolved. 
Accordingly,  the  development  of  the  automated  management  method  of  students'  competences  formation  is 
viewed  to  be  a vital  target. 


2.  COMPETENCES  FORMATION 
2.1  Ontology  of  a Physical  Task 

Design  of  a physical  object  model  is  guided  by  object-oriented  paradigm  with  its  emphasis  on 
decomposition.  Each  object  is  viewed  as  an  instance  of  a particular  class.  Concepts  of  physical  task:  object, 
physical  phenomena,  physical  value,  formula  of  the  law  or  the  physical  quantity  determination.  These  terms 
are  defined  as  classes.  Classes  (concepts)  and  relationships  between  classes  form  the  conceptual  framework 
of  a physical  task,  or  ontology.  Ontology  of  the  physical  task  is  formally  described  by  tuple  type  <C,  I,  L,  P, 
A,  F,  G>,  where  C — concepts  (classes),  I - copies,  L-  dictionary:  EC  u LPu  LI,uLA,  P:  VxV  - relations, 

where  V=C  u U{v|  v £ C or  v £ I},  peP,  p(vi,vj):  Vi  -domain,  Vj  -range.  A:  VxLA  - attributes,  F:  EC— >C, 
G:LP^P.  In  Figure  3 the  part  of  classes’  hierarchy  and  descriptive  logic  mles  are  shown. 

The  Semantic  Web  of  key  concepts  of  the  physical  task  is  represented  as  a graph.  In  Figure  4 the  part  of 
physical  task  ontology  with  relations  is  presented.  Concepts  "object  of  study"  and  "physical  phenomenon"  in 
the  task  are  fundamental.  The  object  of  study  is  involved  in  a physical  phenomenon.  For  example,  the 
material  point  moves,  i.e.  is  involved  in  the  mechanical  motion. 

The  theoretical  material  required  by  solving  the  function  is  included  into  the  ontology  of  a physical  task 
(Kravets  A.G.  & Titova  O.V.,  2011).  The  obtained  functions  are  now  task-doing  competences,  which  are  to 
be  formed  in  students. 

Aiming  at  the  automation  of  competence  level  control  process  there  have  been  set  the  tasks  to  control  the 
system  of  practically  oriented  teaching  in  the  field  of  natural  science,  within  the  framework  of  which  there 
has  been  offered  an  idea  of  mathematical  controlling  the  level  of  students'  competences  (Titova  O.V.  and 
Kravets  A.G.,  2013).  As  a result  of  solving  the  problem  of  the  competence  level  control  there  has  been 
obtained  an  optimal  teaching  mode.  It  is  characterised  by  gradual  increase  in  the  competence  level  during  the 
study  process  (Kravets  A.G.  & Titova  O.V.,  2012). 
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1.  x:  ClassOfObject  ^ object 

2.  y:  ClassOfphysicaLpbenomenon^  physical  phenomenon 

3.  Bobject  ABphysical  phenomenonA  3 invo/ve</_iw (object,  physical 
phenomenon)  a ^described  (physical  phenomenon,  formula)^ 
formula  3 object  a Bphysical  phenomenon  a3  invoZverf_in(object, 
physical  phenomenon)  a ^characterized  (physical  phenomenon, 
physical  quantity)^  physical  quantity 

4.  3 formula  a 3Cowfaiws_a_vaZ«e(formula,  physical  quantity)^ 
physical  quantity 

5.  3 physical  quantity  a 3Contained_in ^ortnaZa (physical  quantity, 
formula)^  formula 


Figure  3.  Hierarchy  of  classes  and  descriptive  logic  rules. 


Figure  4.  Part  of  physical  task  ontology. 

Ontology  a physical  task  it  is  created  in  the  Protege  4.1  software  supporting  format  RDF/OWL.  In 
ontology  the  classes  (concepts)  hierarchy  of  the  physical  task  solution  on  a theme  “Uniform  movement  of  a 
material  point”  is  presented.  The  set  of  the  RDF-statements  received  as  a result  of  a relations  establishment, 
forms  oriented  graph  in  which  tops  are  classes  and  copies,  and  edges  are  marked  by  attitudes. 


2.2  Formation  of  the  Competence 


For  automated  control  of  the  competence  level  it  is  suggested  that  they  perform  a model  of  a student  as  a set 
of  many  competences:  = / Cj,C2,  ■■■,€„}, 

where  C,  - i-th  the  tasks  solution  competence. 

With  the  help  of  the  worked  out  method  of  presenting  a range  of  competences  there  have  been  defined 
certain  connections  between  the  competences.  It  are  represented  the  tasks  solution  competence  in  the  form  of 
sets:  Ci={ki,  k2};  C2={k2,  ky,  kg,  k^,  k^};  C3={ki,  ks};  C4={k2,  k3};  Cs={k3,  kio};  C6={k5};  C7={k4}; 

C8={k6};  C9={kg,  kq};  Cio={kn};  Cn={ki2};  Ci2={k7_ki4}, 

where  C,  - competences,  kj.-  elements  of  a theoretical  material  (packages  of  knowledge). 

For  controlling  the  level  of  the  competences  there  has  been  worked  out  a method  of  the  competence  level 
evaluating  and  developing  an  individual  teaching  strategy  (Titova  O.V.  and  Kravets  A.G.,  2013). 

The  level  of  student  competences  is  evaluated  with  the  help  of  software  developed  in  ASP.NET  MVC  3 
RTM.  It  allows  to  evaluate  the  competences  level  and  to  build  up  an  individual  strategy  of  the  educational 
process  (Figure  5). 

8 10  1 4 11  2 3 5 7 6 9 12 


Figure  5.  Individual  teaching  strategy. 


Criterion  of  involving  order  of  competence  to  the  learning  path:  max  s, , i=l,2,.  ..,n 
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, where  n-number  of  competencies,  Ikj  - grade  for  j-th  package  of  knowledge;  Wy  - 
reentrance  of  j-th  package  of  knowledge  in  i-th  competence. 


If  the  deviation  of  the  competence  level  from  the  target  is  non-zero,  then  the  path  includes  the 
competence.  For  each  knowledge  package  the  tasks  set  is  defined. 

To  train  the  competence  the  special  software  for  intellectual  support  of  optimal  solution  of  analytical 
physical  tasks  was  developed  (Figure  6).  The  code  is  written  in  Java  using  Eclipse  Classic  3.7.2  Indigo  IDE. 
The  program  features  a graphical  user  interface  as  close  as  possible  to  the  form  of  the  solution  in  the  student 
notebook.  To  handle  the  ontology  Protege-OWL  libraries  are  used. 


Task 

description 


Task  text 


Final 

equations 


Figure  6.  Software  for  intellectual  support  of  optimal  solution  of  analytical  physical  tasks. 


2.3  E-learning  of  Physics  Tasks  Decision  Competences  Formation 


During  last  4 years  the  pupils  and  students’  e-leaming  of  physics  tasks  decision  competences  formation  were 
performed.  The  Internet  Consulting  Center  (ICC)  was  organized  in  2004  and  the  new  method  was 
implemented  from  2009. 

The  ICC  education  is  commerce  based.  Without  changing  of  the  payment  amount  it  was  reduced  the 
training  time  by  15%  thanks  to  the  implementation  of  developed  methods  and  programs,  incomes  increased 
by  17,6%.  Number  of  ICC  site  visitors  is  increased  too  (Figure  7).  We  compared  the  average  level  of 
training  at  the  groups  with  traditional  and  experimental  (with  proposed  methods)  ways  of  training  (Figure  8). 


The  dynamics  of  growth  in  the  number  of  site  visitors 
from  2004  to  2012 


Level  of  traning 


Figure  7.  Number  of  ICC  site  visitors 
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Figure  8.  The  diagram  of  training  level 
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The  level  of  training  is  higher  at  the  experimental  groups.  The  results  of  new  methods  implementation  for 
students:  training  quality  increased  by  14,2%;  training  time  decreased  by  15%. 


3.  CONCLUSION 

The  designed  method  of  the  automated  students'  competences  formation  differs  from  the  known  methods  in: 

■ The  ontological  model  of  a physical  task  has  been  the  first  to  be  developed.  Ontology  of  a physical  task 
is  a field  of  knowledge,  which  includes  both  basic  data  of  the  task  and  the  desired  quantity.  Solution  to 
any  task  is  included  into  this  field  of  knowledge. 

■ There  has  been  proposed  a new  method  of  the  task-doing  process  formalization.  It  differs  from  the 
known  ones  due  to  the  carried  out  the  detailed  task-doing  process  and  the  defined  logical  conclusion  for 
completing  the  task  from  the  basic  data  using  ontology  and  descriptive  logic. 

■ Software  for  intellectual  support  of  optimal  solution  of  analytical  physical  tasks  which  uses  an  ontology 
of  physical  tasks  has  been  developed.  The  code  is  written  in  Java,  development  environment  Eclipse 
Classic  3.7.2  Indigo. 

■ As  a result,  learning  formation  of  competences  of  solving  physical  tasks  found  that  the  level  of  training 
in  the  experimental  group  is  higher  by  14.2%  than  in  the  control  group.  The  implementation  of  methods 
and  software  has  reduced  training  time  by  15%.  ICC  has  received  economic  benefits  - increased 
revenue  by  17.6%.  The  effectiveness  of  the  system  of  practice-based  learning  is  higher  in  education 
system  with  the  solving  tasks  competences  formation. 

This  research  proposes  the  innovative  methodology  that  includes  ontology  of  physics  tasks  and  builds  the 
competences  formation  system  to  assist  teachers  to  create  the  e-leaming  practice-oriented  courses. 
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ABSTRACT 

Interaction  and  community  building  are  essential  elements  of  a well  functioning  online  learning  environment,  especially 
in  learning  environments  based  on  investigative  learning  with  a strong  emphasis  on  teamwork.  In  this  paper,  practical 
solutions  covering  quality  criteria  for  interaction  in  online  education  are  presented  for  a simple  implementation  using 
standard  generally  available  tools.  The  solutions  are  evaluated  on  the  basis  of  student  experiences  reported  in  writing  in 
response  to  a qualitative  survey  given  to  adult  university  business  students.  In  student  responses,  the  importance  of  an 
active  online  presence  hy  instructors  was  emphasised,  as  well  as  the  availability  of  team  meeting  platforms.  Blogs  were 
seen  as  an  effective  way  to  support  learning.  Face-to-face  interaction,  both  between  students  and  between  students  and 
instructors,  was  seen  as  less  important  than  expected. 


KEYWORDS 

Investigative  learning,  e-leaming,  blended  learning,  interaction,  online  presence,  blogs 


1.  INTRODUCTION 

In  this  paper,  a description  of  solutions  for  interaction  in  an  online  learning  environment  for  adult  business 
students  in  higher  education  will  be  given.  The  implementation  is  based  on  a learning  model  using 
investigative  learning.  The  focus  is  on  online  solutions,  although  the  implementation  itself  is  a blended 
learning  environment.  The  solutions  will  be  evaluated  on  the  basis  of  student  experiences  and  views  collected 
from  two  different  student  groups.  Some  background  on  the  learning  model  applied  here  will  be  given  first, 
as  the  learning  model  itself  imposes  demands  on  the  implementation. 

Fitting  studying  with  working  and  family  life  is  often  a struggle  for  adult  students.  Efficient  and 
appropriate  learning  environments  are  therefore  especially  important  in  adult  implementations.  When 
students  are  expected  to  do  a lot  of  group  work  and  learn  in  a community,  facilitating  interaction  is 
particularly  essential.  The  solutions  that  work  with  adult  students  can,  with  some  consideration,  be  applied  to 
other  student  groups  as  well. 


2.  LEARNING  BY  DEVELOPING 

The  learning  method  called  Learning  by  Developing  is  defined  as  investigative  learning  applied  to 
development  projects  (Rauhala,  2006).  Hakkarainen  et  al.  (2004,  17)  claim  that  learning  is  at  best  an 
investigative  process.  This  means  not  only  analytical  information  processing  and  assessment,  but  also 
practical  experimenting  and  learning  from  experience.  Wenger  (1998,  73)  proposes  that  learning  and 
expertise  are  transmitted  through  unofficial  communities  of  practise.  These  communities  have  common 
goals,  exist  at  work  or  other  environments,  but  their  members  are  not  necessarily  aware  of  this  communal 
structure. 
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One  aspect  of  learning  in  a community  is  the  sharing  of  the  learning  process  and  its  outcomes.  Sharing 
one’s  expertise  provides  a forum  for  testing  that  expertise  and  justifying  one’s  viewpoints  (Hakkarainen  et 
ah,  186).  In  collaborative  problem  solving  the  participants  typically  adopt  different  roles  and  serve  their 
different  viewpoints  to  each  other,  thus  deepening  their  understanding  of  the  problem  (Miyake,  1986).  To  the 
individual,  the  community  or  the  group  provides  scaffolding,  permitting  the  individual  to  perform  tasks  more 
demanding  than  would  be  possible  for  them  by  themselves  (Brown,  1993,  191). 

In  authentic  situations,  contrary  to  traditional  educational  environments,  there  are  often  no  known 
solutions  to  the  problems.  The  solutions  are  always  the  result  of  a more  thorough  process,  including  the 
definition,  analysis  and  description  of  the  problem,  as  well  as  the  choice  of  suitable  methods  to  be  applied 
(Franti  & Pirinen,  2005,  37).  Most  of  these  steps  will  be  omitted  when  the  problem  is  defined  and  proposed 
by  the  teacher.  With  this  approach,  it  is  hoped  that  flexible  and  intuitive  problem  solving  will  be  achieved, 
not  restricted  to  specific  situations  and  contexts.  This  fulfils  an  important  requirement  of  professional 
expertise,  as  defined  by  Tynjala  (1999,  160-161). 

The  key  concepts  behind  the  Learning  by  Developing  model  are  authenticity,  companionship,  an 
experience-based,  investigative  approach  and  creativity  (Raij,  2007).  Authenticity  is  achieved  through  real- 
life  development  projects  serving  outside  partners.  In  practice,  this  means  that  students  mostly  study  in 
teams,  carrying  out  projects.  Companionship  means  shared  working,  learning  and  responsibility.  Practical 
experience  is  the  medium  through  which  learning  and  new  knowledge  is  reflected  upon  and  identified.  The 
investigative  approach  is  the  way  in  which  learning  is  organised,  as  described  above.  Creativity  is  seen  in  the 
adoption  of  new  solutions  and  working  methods.  An  online  environment  can  be  a good  platform  to  foster 
investigative  learning.  Reeves  (1999)  describes  how  the  cognitive  load  of  the  learner  can  be  deepened  by 
online  tools,  with  the  teacher  acting  in  the  role  of  coach.  This  corresponds  exactly  to  the  roles  of  student  and 
teacher  in  Learning  by  Developing. 


3.  ONLINE  INTERACTION  AND  COMMUNITIES 

A sense  of  community  and  social  presence  has  been  widely  acknowledged  to  be  a factor  in  enhancing  both 
the  quality  of  learning  and  the  motivation  to  study  (Haythornthwaite  and  Andrews,  2011,  111-112).  In 
establishing  an  online  learning  community,  different  types  of  interaction  play  an  essential  role. 

Berge  (2002)  reminds  that  interaction  cannot  be  viewed  only  for  its  own  sake,  but  in  context  with  the 
methods  and  systems  available  in  the  given  situation.  That  is  also  the  approach  here;  the  aim  is  not  to  provide 
a general  framework  but  rather  to  investigate  the  possibilities  and  improvement  needs  in  a given 
environment.  The  implementation  follows  the  same  design  as  that  presented  by  Berge  (2002):  aligning 
learning  goals,  activities  and  evaluation,  with  a learner-centred  approach,  designed  to  provide  interaction 
between  active  and  reflective  learning.  For  active  learning  to  be  encouraged,  interaction  needs  to  be  activity- 
based  (Hirumi,  2002). 

Hirumi  (2002)  identifies  three  levels  of  interaction:  the  interaction  of  the  learner  with  himself  (level  1)  or 
with  human  and  non-human  resources  in  the  learning  environment  (level  2).  The  third  level  of  interaction  is  a 
meta-level  and  describes  the  interaction  of  the  learner  with  the  e-leaming  strategy  enforced.  Although  the 
third  level  is  inevitably  present,  this  paper  focuses  on  level  2 interactions.  The  second  level  is  comprised  of 
the  human  interaction  between  students  and  of  students  with  teachers,  but  also  of  the  interaction  of  students 
with  the  content  and  interface  of  the  learning  environment.  Other  interactions  are  also  possible,  e.g.  with 
people  outside  the  learning  environment,  such  as  experts  or  project  partners.  Second  level  interaction  is  the 
most  obvious  one  because  it  is  related  to  the  most  visible  parts  of  online  learning  if  there  is  to  be  any 
interaction,  but  third  level  planning  is  just  as  important,  if  not  more  so,  because  it  is  the  level  that  is  often 
neglected  when  moving  instruction  online  from  traditional  implementations  (Hirumi,  2002).  The  process  of 
planning  and  evaluation  on  online  implementations  follows  quite  closely  the  procedure  proposed  by  Hirumi 
(2002),  with  learning  goals  as  a starting  point. 
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4.  QUALITY  CRITERIA  IN  ONLINE  INSTRUCTION  AND  THEIR 
APPLICATION  TO  INTERACTION  IN  THE  LEARNING 
ENVIRONMENT 

Karjalainen  has  identified  a list  of  goals  to  strive  for  in  online  education,  based  on  Herrington’s  three  types  of 
approaches  to  quality  in  online  education:  pedagogies,  resources  and  delivery  strategies  (Herrington  et  al, 
2001).  Similar  ideas  are  presented  by  Berge  (2002).  Karjalainen  establishes  the  following  check  list: 

1 . A clear  and  organised  structure.  Materials,  assignments  and  activities  should  be  structurally  clear  and 
easy  to  contextualise. 

2.  Goal-oriented  learning.  Students  should  know  the  learning  goals  and  be  able  to  relate  them  to  their 
own  goals. 

3.  Authentic  activities  based  on  real  life  situations.  Learning  tasks  should  be  inquiry  and  problem-based 
and  provide  a meaningful  context  for  students  to  develop  their  skills. 

4.  Appreciation,  connecting  and  application  of  knowledge.  The  students’  previous  knowledge  should  be 
recognised  and  linked  to  the  construction  of  new  knowledge  and  its  application  to  practice. 

5.  Emphasis  on  collaboration  and  interaction.  The  online  material  and  activities  should  encourage 
collaboration  and  interaction  between  students,  as  well  as  between  students  and  teachers. 

6.  Activation.  The  material  and  activities  should  activate  learning. 

7.  Learner-centred  environment.  Design  and  implementation  should  be  centred  on  learning,  rather  than 
teaching. 

8.  Integrated  and  well-timed  assessment  and  feedback.  Assessment  should  support  the  learning  process 
and  feedback  should  be  integrated  into  the  learning  activities. 

9.  Self-directed  learning.  The  students  should  be  able  to  navigate  the  learning  process  independently  at 
their  own  pace.  (Karjalainen) 

The  pedagogical  criteria  were  used  to  design  an  online  environment  as  part  of  a blended  learning 
implementation.  All  criteria  were  analysed  and  broken  down  into  factors  that  could  be  used  in  the  online 
environment  to  promote  the  above  goals.  As  the  focus  of  this  paper  is  on  interaction,  only  the  fifth  criterion 
will  be  discussed  here.  The  following  solutions  were  adopted  to  respond  to  the  need  for  engaging  students  in 
collaboration  and  interaction. 

Establishing  an  online  presence  in  the  environment.  All  teachers’  photographs  are  provided  with  study 
unit  presentations.  The  students  are  asked  to  upload  their  own  photographs  on  a presentations  page. 
Photographs  are  also  linked  to  an  online  profile,  which  ensures  them  being  appended  to  each  message  a user 
sends  in  the  environment,  whether  it  be  on  a bulletin  board  or  as  a comment  to  blog  posts.  In  addition  to  this, 
the  teachers  attempt  to  show  a constant  presence  online  by  commenting  on  students’  posts  and  answering 
messages  when  needed. 

Teamwork.  Most  assignments  are  project-based  teamwork  activities,  forcing  the  students  to  find  ways  to 
work  together. 

Teamwork  infrastructure.  Student  teams  have  their  own  folders  in  the  online  environment,  with  tools 
provided  to  facilitate  interaction,  including  a discussion  board  and  a video  chat  room.  Additional  tools  are 
provided  when  possible  if  students  express  a wish  to  use  them. 

Study  unit  blogs.  Blogs  are  used  to  provide  a forum  for  content  and  activity-related  discussion  (as 
opposed  to  practical  arrangements,  for  which  an  ordinary  discussion  board  is  used).  Students  can  write  blog 
posts  asking  questions  or  seeking  clarification  for  assignments.  They  can  share  ideas  or  help  each  other  with 
tips  or  explanations.  Teachers  use  the  blog  to  give  feedback  on  assignments  and  to  give  tips  or  guidance  on 
activities.  They  also  write  posts  on  face-to-face  sessions.  This  serves  the  purpose  of  reminding  students  of  the 
important  ideas  discussed  there  and  of  preparing  them  for  the  next  teaching  sessions. 

Peer  assessment.  Making  reports  visible  to  all  students  and  asking  the  students  to  give  feedback  on  other 
teams’  reports  is  a way  to  involve  the  students  more  closely  in  working  together  and  helping  each  other. 

Virtual  group  sessions.  An  online  meeting  environment  is  used  for  online  lectures  and  guidance 
meetings. 

5.  STUDENT  EXPERIENCES  ON  INTERACTION 

Experiences  and  comments  were  gathered  from  students  from  two  groups:  first  and  third  year  students 
studying  Business  Management.  The  groups  have  face-to-face  teaching  about  once  a month  (one  evening  and 


358 


lADIS  International  Conference  e-Learning  2013 


two  days  at  a time),  while  the  students  work  full  time  and  do  most  of  the  studying  online.  Thirteen  students 
participated.  They  were  asked  to  write  freely  about  their  thoughts  on  online  interaction  and  more  specifically 
about  teamwork  and  about  the  tools  mentioned  above.  The  answers  were  gathered  through  an  online 
questionnaire,  containing  mostly  open  questions. 

Some  background  questions  were  asked  to  establish  the  students’  orientation  towards  online  studies  and 
the  learning  model  used  in  this  context.  Most  of  the  students  described  themselves  as  being  comfortable  with 
studying  online  and  with  doing  teamwork,  although  one  third  stated  that  they  would  prefer  to  study  by 
themselves  rather  than  with  others. 

5.1  Experiences  about  Online  Teamwork  Tools 

The  students  thought  that  both  advantages  and  disadvantages  of  teamwork  are  magnified  in  online  studying. 
The  advantages  associated  with  teamwork  were  the  support  given  by  teammates,  the  benefit  of  having 
different  views  into  the  same  topic  and  the  possibility  to  share  the  work  load  with  others.  There  was  mainly 
one  negative  aspect  brought  forward:  the  different  goals  of  team  members  and  different  degrees  of 
commitment  to  the  project  and  to  teamwork.  Students  found  it  difficult  to  find  ways  to  control  teamwork  and 
influence  the  team’s  results  if  other  team  members  had  different  views.  They  also  complained  about  the 
difficulty  to  agree  on  timetables. 

When  asked  about  the  tools  supporting  teamwork,  the  students  were  almost  unanimous  in  praising  the 
virtual  meeting  rooms  and  chat  rooms  as  useful.  The  frequency  of  using  them  differed,  however,  between 
teams,  from  weekly  to  infrequently.  Some  had  been  frustrated  by  technical  difficulties  (usually  related  to 
microphone  use)  and  had  moved  to  other  tools  not  provided  by  the  university,  such  as  phone  conferences  or 
other  chat  rooms.  Several  respondents  stressed  that  speech  was  not  a necessary  element  in  team  meetings  and 
they  appreciated  written  communication  for  its  precision  and  unequivocal  documentation.  The  students  noted 
that  they  frequently  used  other  communication  platforms  such  as  Facebook  for  teamwork. 

The  students  were  divided  on  the  usability  of  virtual  meeting  platforms  for  group  sessions.  Some  had 
positive  experiences  of  them,  while  others  thought  virtual  meeting  rooms  were  only  suited  for  a few  people  at 
a time.  The  importance  of  planning  the  session  carefully  and  having  a clear  agenda  was  emphasised  by  many. 

5.2  Blogs 

One  of  the  most  useful  ways  of  interacting  online  seems  to  be  the  study  unit  blog.  The  students  said  that  the 
most  important  online  interaction  is  content-related  and  the  blog  was  well  suited  to  this.  They  wished  that  all 
questions  would  be  asked  through  the  blogs  to  make  them  visible  to  all  and  that  everyone  would  share  their 
thoughts  on  the  subjects  studied,  as  they  appreciated  the  different  views.  The  most  useful  functions  in  the 
blogs,  according  to  the  students,  were  tips  and  clarifications  to  assignments,  the  possibility  to  ask  questions 
and  blog  posts  about  past  teaching  sessions.  Also  named  were  feedback  posts  about  assignments  and 
discussions  about  subject  matters  with  other  students. 

5.3  Interaction  between  Students  and  Teachers 

When  asked  what  kind  of  interaction  the  students  wished  to  have  with  teachers,  they  said  that  teachers  should 
above  all  be  regularly  present  and  active.  Answering  questions  promptly  is  important,  as  when  this  does  not 
happen  the  students  have  to  start  sending  questions  by  e-mail,  thus  removing  the  benefit  of  having  the 
answers  visible  to  all.  The  general  opinion  seemed  to  be  that  face-to-face  or  even  virtual  oral  interaction  with 
the  teacher  was  not  necessary  if  assignments  and  instructions  were  clearly  formulated  and  feedback  was  well 
given.  The  importance  of  precise  and  useful  feedback  came  up  repeatedly  in  other  contexts  as  well,  as  did  the 
importance  of  well  planned  activities  and  assignments  and  information  given  in  a timely  fashion. 

The  students  also  expressed  some  more  general  views  about  the  learning  environment  and  pedagogy. 
Several  students  wrote  that  in  a blended  learning  setting,  face-to-face  teaching  sessions  should  really  have 
some  added  value  and  not  be  held  just  for  the  sake  of  meeting  and  discussing  things.  This  was  a little 
surprising,  as  there  has  been  an  assumption  that  face-to-face  meetings  should  include  some  time  for 
discussion  and  less  formal  interaction.  The  students’  wish  for  structured  and  “efficient”  teaching  runs  counter 
to  the  generally  accepted  view  that  constructive  learning  benefits  from  the  sharing  of  views  and  at  least  some 
degree  of  freedom  in  interaction. 
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6.  CONCLUSIONS 

The  lessons  drawn  from  this  are  twofold.  On  one  hand,  good  planning  and  communication  of  learning  goals 
and  feedback  are  - unsurprisingly  - important.  On  the  other  hand,  the  importance  of  direct  interaction 
between  teachers  and  students  is  less  important  than  was  originally  thought.  Hirumi  (2002)  warns  of  the 
overload  to  instructors  caused  by  poorly  designed  interaction  or  too  much  planned  learner-instructor 
interaction.  These  results  suggest  that  this  overload  could  be  avoided  with  adequate  planning,  without  too 
much  risk  of  students  feeling  a lack  of  interaction.  This  frees  the  teachers  to  possibly  use  more  resources  for 
the  non-interactive  part  of  online  learning  facilitation,  such  as  planning  appropriate  activities  and  working  on 
assignments  and  material. 

However,  the  importance  of  a constant  online  presence,  even  through  seemingly  small  and  trivial  actions, 
cannot  be  underestimated.  Slightly  surprising  - and  delightful  - is  the  positive  attitude  of  the  students  to 
between-students  interaction,  as  this  has  previously  been  a topic  of  concern.  One  subject  not  much  addressed 
here  is  the  building  of  an  online  community.  The  existence  of  a community  is,  both  in  teachers’  and  students’ 
eyes,  a fact  from  a very  early  point  in  time.  Haythornthwaite  and  Andrews  (2001,  120-121)  support  the  view 
that  face-to-face  meetings  foster  this  feeling  of  community.  This  has  also  been  the  assumption  in  the  case 
described  here,  but  some  of  the  student  comments  reported  above  contradict  this.  How  the  online  community 
and  its  building  are  perceived  by  students  and  what  factors  influence  community  spirit  the  most  are  questions 
that  would  benefit  from  a more  systematic  investigation. 

The  work  continues,  as  solutions  for  the  best  online  interaction  will  continue  to  be  developed  and  tested. 
The  other  dimensions  of  online  instruction  quality  deserve  similar  detailed  evaluation. 


REFERENCES 

Berse  Z.  2002.  Active,  Interactive  and  Reflective  eLearning.  In  The  Quarterly  Review  of  Distance  Education,  Vol.  3, 
No.  3,  pp  181-190. 

Brown  A.  L.,  Ash  D.,  Rutherford  M.,  Nakagawa  K.,  Gordon  A.,  Campione  J.,  1993.  Distributed  expertise  in  the 
classroom.  In  Salomon  G.  (ed.).  Distributed  cognitions:  Psychological  and  educational  considerations.  Cambridge 
University  Press,  Cambridge,  UK. 

Franti  M.  & Pirinen  R.,  2005.  Tutkiva  oppiminen  integratiivisissa  oppimisympdristdissd  BarLaurea  ja  REDLabs.  Laurea 
University  of  Applied  Sciences,  Helsinki,  Finland. 

Hakkarainen  K.,  Lonka  K.,  Lipponen  L.,  2004.  Tutkiva  oppiminen.  Jdrki,  tunteet  ja  kulttuuri  oppimisen  sytyttdjind. 
WSOY,  Porvoo,  Finland. 

Haythornthwaite  C.  & Andrews  R.,  2011.  E-learning  Theory  & Practice.  Sage,  Chippenham,  UK. 

Herrington  A.,  Herrington  J.,  Oliver  R.,  Stoney  S.,  Willis  J.,  2001.  Quality  guidelines  for  online  courses:  the  development 
of  an  instrument  to  audit  online  units,  http://ro.ecu. edu.au/cgi/viewcontent.cgi?article=5814&context=ecuworks 

Hirumi  A.,  2002.  A Framework  for  Analyzing,  Designing  and  Sequencing  Planned  eLearning  Interactions.  In  The 
Quarterly  Review  of  Distance  Education.  Vol.  3,  No.  2,  pp  141-160. 

Karjalainen  K.  Laadukasta  verkko-oppimateriaalia  tuottamassa. 

http://www.vopla.fi/tiedostot/Laatukasikirja/Oppimateriaali/laadukasta%20verkko- 
oppimateriaalia%20tuottamassa_final.pdf 

Miyake  N.,  1986.  Constructive  interaction  and  the  iterative  process  of  understanding.  In  Cognitive  Science,  Vol.  10, 
pp  151-177. 

Raij  K.,  2007.  Learning  by  Developing.  Laurea  University  of  Applied  Sciences,  Vantaa,  Finland. 

Rauhala  P.,  2006.  Pedagoginen  ajattelu  Laureassa.  In  Erkamo  M.,  Haapa  S.,  Kukkonen  M.-L.,  Lepistb  L.,  Pulli  M.,  Rinne 
T.  (ed.  ) Uudistuvaa  opettajuutta  etsimdssd.  Laurea  University  of  Applied  Sciences,  Helsinki,  Finland. 

Reeves  T.,  1999.  A Model  of  the  Effective  Dimensions  of  Interactive  Learning  on  the  World  Wide  Web.  Athens,  GA, 
USA.  The  University  of  Georgia.  http://treeves.coe.uga.edu/WebPaper.pdf 

Tynjaja  P.,  1999.  Konstruktivistinen  oppimiskasitys  ja  asiantuntijuuden  edellytysten  rakentaminen  koulutuksessa. 
In  Eteldpelto  A.  & Tynjdld  P.  (ed.),  Qppiminen  ja  asiantuntijuus.  Tydeldmdn  ja  koulutuksen  ndkbkulmia.  WSOY, 
Porvoo,  Finland. 

Wenger  W.,  1998.  Communities  of  practice:  Learning,  meaning  and  identity.  Cambridge  University  Press,  Cambridge, 
UK. 


360 


lADIS  International  Conference  e-Learning  2013 


USABILITY  ASSESSMENT  OE  E-CAEE  OPERATIONAL 
MANAGEMENT  SIMULATION  GAME 


Chiung-sui  Chang  and  Ya-Ping  Huang 

Department  of  Educational  Technology,  Taipei,  Taiwan 


ABSTRACT 

To  ensure  the  quality  of  digital  simulation  game,  we  utilized  the  usability  evaluation  heuristic  in  the  design  and 
development  processes  of  e-cafe  operational  management  game-hased  learning  material  for  students.  The  application  of 
usability  evaluations  during  this  study  is  described.  Additionally,  participant  selection,  data  collection  and  analysis;  and 
results  relevant  to  usability  portions  of  this  study  are  presented. 
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1.  INTRODUCTION 

Nowadays  college  and  university  students  have  never  experienced  a world  without  computers.  Since  digital 
simulation  game  is  designed  more  visual,  interactive,  and  focused  on  problem-solving  (Pasin,  & Giroux, 
2011).  It  is  more  attractive  for  students  to  learn  the  complex  knowledge  via  digital  simulation  game  for 
college  students.  Therefore,  this  study  employed  design-based  research  to  examine  the  design  and 
development  of  an  educational  game  simulating  the  daily  operational  management  of  an  “e-cafe”.  This  study 
employed  the  principle  of  “usability”  to  determine  whether  this  material  corresponds  to  the  above  principle 
and  whether  it  can  meet  the  expectations  of  learners  as  an  education  resource. 


2.  LITERATURE  REVIEW 


2.1  Digital  Simulation  Game 

Digital  simulation  games  are  used  for  education  and  training  in  a wide  range  of  fields  including  the  military, 
medical  services,  and  scientific  experiments.  Unlike  digital  games,  simulation  games  are  usually  less 
associated  with  fantasy.  Simulation  games  involve  players  assuming  roles  through  which  they  seek  to  resolve 
issues  or  accomplish  missions  (Charsky,  2010).  Digital  simulation  facilitates  learning  within  created 
scenarios  while  enjoying  gameplay  that  greatly  resembles  situations  in  the  real  world.  Following  a series  of 
game  activities,  learners  respond  to  various  scenarios  while  achieving  learning  objectives.  Researchers  have 
indicated  that  digital  games  with  educational  functions  can  exert  a positive  effect  on  learners  with  regard  to 
motivation  and  performance  (Tompson  & Dass,  2000,  Bai,  Pan,  Hirumi,  & Kebritchi,  2012). 

2.2  Usability  Testing 

Usability  can  reveal  whether  a given  product  serves  the  needs  of  users  and  whether  the  product  can  be 
operated  intuitively.  Nielson  (1993)  proposed  five  indicators  for  the  evaluation  of  usability:  learnability, 
efficiency,  memorability,  and  error  and  satisfaction.  These  indicators  can  be  used  to  determine  whether  users 
are  able  to  browse  a webpage  without  facing  obstacles,  whether  the  page  can  be  accessed  intuitively,  and 
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whether  the  webpage  provides  information  sufficient  to  satisfy  the  needs  of  users.  The  International 
Organization  for  Standardization  (ISO)  pointed  out  in  1998  (ISO  9241-11)  that  “Usability’  refers  to  the 
extent  to  which  a product  can  be  used  by  specified  users  to  achieve  specified  goals  with  effectiveness, 
efficiency  and  satisfaction  in  a specified  context  of  use.”  The  most  common  methods  employed  in  the 
evaluation  of  usability  include  heuristic  evaluation  and  usability  testing.  This  study  recruited  subjects  with 
the  aim  of  revealing  user  problems,  evaluating  the  usability  of  the  educational  simulation  game,  and 
providing  the  basis  from  which  to  revise  teaching  materials  in  the  future. 


3.  METHODOLOGY 

Our  evaluation  of  usability  involved  both  quantitative  and  qualitative  methods,  the  collection  of  data  through 
questionnaires,  observation  and  interviews,  and  the  compilation  of  opinions  and  suggestions  for  the  design 
and  usability  of  the  e-cafe  simulator.  This  study  will  expound  the  subjects,  testing  processes,  and  research 
tools  as  follows. 

3.1  Subjects  and  Testing  Processes 

Ten  participants  were  invited  to  take  part  in  the  testing  of  a simulation  game  with  regard  to  usability.  All 
participants  were  college  students  at  a male/  female  ratio  of  3:2  and  all  were  participating  in  this  type  of 
exercise  for  the  first-time.  Through  feedback  from  the  subjects,  we  came  to  realize  the  needs  of  users  and 
revised  the  simulation  game  accordingly.  Tests  and  interviews  were  conducted  with  regard  to  the  content  and 
usability  of  the  game. 

3.2  Testing  Tools 

Each  participant  was  observed  one-on-one  in  the  usability  testing  process  and  Morae  software  was  applied  to 
videotape  the  entire  process.  Prior  to  testing,  the  objectives  of  the  evaluation  were  outlined  for  participants, 
who  were  then  requested  to  sign  a consent  letter.  Throughout  the  testing  process,  the  participants  were  closely 
observed  and  provided  objectives  (tasks)  in  accordance  with  their  progress.  Subjects  were  expected  to 
accomplish  each  phase  of  their  tasks  in  accordance  with  the  statements  provided  by  observers.  Participants 
were  expected  to  undertake  eleven  missions  and  observers  kept  a close  eye  to  verify  whether  they  were  able 
to  complete  them  on  their  own. 

3.2.1  Questionnaire  on  Task  Difficulty 

The  eleven  tasks  in  this  study  were  designed  to  determine  whether  the  subjects  were  able  to  complete  the 
tasks  they  were  assigned  in  the  digital  simulation  game.  Questions  were  ranked  on  a scale  of  1 to  5 with  1 
representing  “very  easy”  and  5 representing  “very  difficult”. 

3.2.2  Questionnaire  on  User  Experience 

This  study  adopted  the  TAM  model  to  measure  the  responses  of  participants  following  completion  of  the 
simulation  game.  This  questionnaire  comprised  a total  of  22  questions  in  four  dimensions:  attitude,  perceived 
ease  of  use,  intention  to  use,  and  perceived  usefulness.  Questions  were  ranked  on  a scale  of  1 to  5 with  1 
representing  “strongly  disagree”  and  5 representing  “strongly  agree”. 

3.2.3  Interview  Outline 

Following  completion  of  the  usability  test,  interviews  were  conducted  to  determine  the  feelings  of 
participants.  The  four  interview  questions  were  as  follows: 

Ql:  Did  you  encounter  any  problems  when  undergoing  tasks  in  the  game?  How  did  you  resolve  them? 

Q2:  Did  you  better  eomprehend  the  process  of  running  a eafe  after  eompleting  the  “e-eafe”  simulation  game? 
Q3:  Please  outline  the  concepts  related  to  business  administration  that  you  learned  from  the  game? 

Q4:  Do  you  think  the  “e-eafe”  management  simulation  game  helped  you  to  aequire  knowledge  about 
business  administration? 
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4.  RESULTS 


4.1  Result  Analysis  for  the  Questionnaire  on  Tasks  Difficulty 

The  questionnaire  was  meant  to  determine  how  the  subjects  felt  about  each  mission.  For  the  eleven  question 
items,  the  average  score  was  1.8  (SD=0.93).  The  scale  of  difficulty  ranged  from  1 (very  easy)  to  5 (very 
difficult).  The  participants  felt  that  task  8 was  the  most  difficult  to  accomplish  (M=2.7),  followed  by  task  10 
(M=2.2),  task  6 (M=2.1)  and  task  7 (M=2.1).  Task  8 involved  adjusting  the  shift  schedule.  Task  10  required 
that  participants  organize  staff  training  sessions.  Task  6 required  subjects  to  improve  the  quality  of  raw 
materials.  Task  7 required  subjects  to  devise  a marketing  approach  to  expand  the  customer  base.  In  task  5, 
participants  were  required  to  adjust  the  selling  price. 

4.2  Result  Analysis  for  Game  User  Experience 

Questions  were  ranked  on  a scale  of  1 to  5,  with  1 representing  “strongly  disagree”  and  5 representing 
“strongly  agree”.  As  shown  in  Table  4,  the  average  score  was  3.5  (SD=0.86),  suggesting  that  participants  had 
positive  attitudes  toward  the  simulation  game.  In  the  “Attitude”  sub-category,  the  four  questions  averaged  3.7 
(SD=1.01),  implying  that  most  of  the  subjects  enjoyed  the  simulation  game.  In  “Perceived  ease  of  use,”  the 
five  questions  averaged  3.44  (SD=0.88),  which  indicated  that  participants  felt  the  game  was  easy.  In 
“Intention  to  use,”  the  three  questions  averaged  3.9  (SD=0.85),  and  each  sub-question  reached  3.8  or  more, 
indicating  that  subjects  strongly  accepted  the  simulation  game.  The  average  score  for  sub-question  3 was  4.1 
(SD=0.74),  indicating  that  subjects  were  keen  to  participate  in  simulation  games.  In  “Perceived  usefulness,” 
the  five  questions  averaged  3.36  (SD=0.8),  indicating  that  participants  felt  the  digital  simulation  game  is 
useful  for  them. 

4.3  Result  Analyses  of  Interviews 

Interview  analysis  was  meant  to  reveal  the  experience  of  users  and  examine  areas  that  were  not  adequately 
covered  by  questionnaires  or  observation.  Interview  analysis  related  to  the  questions  is  as  follows: 

Ql.  Did  you  encounter  any  problems  when  undergoing  missions  in  the  game?  How  did  you  solve  it? 
Ten  participants  outlined  problems  they  encountered  when  playing  the  simulation  game.  Three  subjects 
did  not  know  that  the  staffer’s  name  could  be  clicked.  Two  of  the  participants  were  unable  to  differentiate  the 
faces  of  staffer  and  felt  that  a failure  to  mention  the  names  of  staffers  made  it  difficult  to  arrange  shift 
schedules.  One  participant  did  not  know  that  the  page  could  be  changed  from  the  counter  to  the  guest  area 
simply  by  clicking  and  kept  wondering  why  the  coffee  machine  could  not  be  clicked  to  make  repairs. 
However,  most  of  these  were  resolved  after  the  subjects  tried  several  times.  Clearly,  the  simulation  game 
requires  the  means  to  explain  all  of  the  game  functions.  Nonetheless,  these  observations  elucidated  the  needs 
and  expectations  of  the  subjects,  such  that  the  game  could  be  improved  in  the  future. 

Q2.  Did  you  better  comprehend  the  process  of  running  a cafe  after  completing  the  “e-cafe” 
simulation  game? 

Nine  out  of  ten  subjects  (except  for  S8)  believed  that  the  simulation  game  helped  them  to  understand  how 
to  manage  a cafe. 

Q3.  Please  outline  the  concepts  related  to  business  administration  that  you  learned  from  the  game? 

All  subjects  felt  that  they  developed  their  skills  related  to  business  administration  from  the  simulation 
game.  These  skills  included  cost  control,  arranging  shift  schedules,  picking  the  location  in  which  to  open  a 
shop,  the  management  of  raw  materials,  equipment  repair  and  maintenance,  profit,  and  procurement. 
Nonetheless,  a number  of  other  opinions  were  also  expressed: 

SI:  It  appears  to  me  that  hands-on  experience  would  be  more  helpful.  S2:  I never  took  any  class  in 
business  administration  before,  so  I cannot  say  for  sure.  As  to  concepts,  I think  the  most  helpful  ideas 
were  about  management,  shift  schedules,  and  the  timing  of  equipment  maintenance/repair.  S4:  I believe 
that  cost  control  and  customer  satisfaction  were  dealt  with  effectively  in  the  game. 
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Q4.  Do  you  think  that  the  e-cafe  management  game  was  of  any  help  to  you  in  acquiring 
knowledge  in  business  administration? 

Eight  subjects  believed  that  the  simulation  game  could  be  used  to  complement  business  administration 
curriculum.  However,  SI  and  S8  disagreed. 

S5:  Yes.  (The  researcher  asked  about  why.)  Ugh,  seems  that  we  can  have  a better  time  in  the  classroom. 
S6:  Yes,  I enjoyed  playing  the  game.  S6  felt  that  if  this  game  was  played  during  class,  they  would  not 
need  to  peruse  reading  materials.  S8:  I have  never  taken  any  classes  in  business  administration; 
however,  I am  left  wondering  if  this  would  be  too  simple  to  be  fit  into  the  real  world.  For  example,  staff 
affairs  appeared  too  simplified.  Raw  materials,  prices,  and  the  quality  of  materials  appeared  more 
authentic. 


5.  DISCUSSION  AND  CONCLUSION 

We  conducted  an  examination  of  the  usability  of  the  “e-cafe  management  digital  simulation  game”  as  well  as 
the  test  results  and  feedbacks  from  the  participants.  Our  aim  was  to  identify  the  areas  corresponding  to  the 
expectations,  requirements,  and  suggestions  of  users  for  use  as  a reference  in  the  development  of  educational 
simulation  games. 

With  regard  to  usability,  every  participant  presented  a positive  attitude  toward  every  dimension  of  the 
e-cafe  simulation  game.  The  mission  observation  and  the  scale  of  difficulty  perceived  by  participants  were 
matched,  as  subjects  encountered  difficulties  in  the  complex  missions  but  felt  that  most  of  the  missions  were 
quite  easy.  Interviews  revealed  that  subjects  regarded  the  game  as  a valuable  tool  in  developing  the  skills 
required  for  shop  management.  We  believe  that  the  reason  for  the  strong  acceptance  of  the  simulation  game 
was  its  overall  design,  which  accurately  represented  a real  coffee  shop.  It  is  our  hope  that  this  study  provides 
a valuable  resource  for  the  development  of  digital  simulation  games  as  an  educational  resource. 
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ABSTRACT 

A system  was  developed  for  automatic  generation  of  problems  and  solutions  for  examinations  in  a university  distance 
course  in  discrete  mathematics  and  tested  in  a pilot  experiment  involving  200  students.  Considering  the  success  of  such 
systems  in  the  past,  particularly  including  automatic  assessment,  it  should  not  take  long  before  such  systems  are  widely 
used  in  higher  mathematics  education.  The  goal  of  this  paper  is  to  encourage  such  development  by  sharing  some  of  the 
experience  that  was  gained  along  the  way  in  the  implementation  and  application  of  such  system. 
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1.  INTRODUCTION 

Automatic  generation  and  grading  of  examinations  is  a promising  development  that  may  potentially  save 
time  and  effort  for  professors  within  the  field  of  technology,  giving  room  for  further  improvements  in  the 
education,  along  with  allocation  of  additional  time  for  discussions  with  the  students.  Such  systems  may  also 
be  used  for  the  generation  of  individualized  examinations,  decreasing  the  risk  for  plagiarism  while  at  the 
same  time  offering  each  student  higher  flexibility  on  examination  dates,  followed  by  decreased  grading  time 
and  increased  grading  accuracy  (Beevers,  2000;  Maple  TA,  2013;  NCES,  2005;  Bead,  2010;  Rasila,  2010). 

To  contribute  to  this  development,  an  automatic  item  generator  called  the  Virtual  Mathematics  Assistant 
(VMA)  was  developed  by  the  author  and  launched  at  Gotland  University  (recently  merged  with  Uppsala 
University,  Sweden),  with  the  ability  to  generate  full  examination  papers  and  solutions  in  a distance  course  in 
discrete  mathematics  for  computer  science  students.  This  generator  is  presently  able  to  create  examination 
papers  covering  2048  courses  (with  69  assignments  per  course  occasion)  or  a total  amount  of  over  100,000 
unique  assignments  within  over  ten  different  areas  of  discrete  mathematics.  The  VMA  system  was  applied 
during  two  consecutive  years  totally  covering  around  200  students.  Compared  to  previous  examinations,  the 
new  ones  succeeded  in  addition  to  reduce  the  grading  time  for  each  student  to  about  one-third  compared  to 
previous  years  without  compromising  the  quality  of  the  course. 


2.  MATERIALS  AND  METHODS 

The  VMA  system  was  developed  in  C-H-  using  Apple  Xcode  (2013)  for  the  generation  of  LaTex-code  and 
TeXShop  (2013)  for  generation  of  the  PDF  examination  documents.  C-H-  is  a common  programming 
language  for  development  of  software  applications.  LaTex  is  the  standard  typesetting  system  for 
mathematical  documents  and  is  often  used  for  generation  of  scientific  papers. 

The  VMA  system  is  designed  to  operate  in  two  primary  modes,  development  and  production.  In  the 
development  mode  the  programmer  is  able  to  focus  on  the  implementation  and  validation  of  a single  model 
or  engine  at  a time  and  to  make  sure  that  each  component  in  the  system  performs  properly.  In  production 
mode,  the  user  currently  sets  a course  index  between  0 and  2047  and  compiles  and  runs  the  VMA 
application,  whereby  twelve  examination  papers  are  generated  in  LaTex-format  plus  twelve  papers  that 
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include  both  the  problems  and  the  solutions  covering  a single  course  in  discrete  mathematics.  In  addition  a 
number  of  examination  papers  with  solutions  are  generated  for  the  students  to  study  in  advance  of 
examinations. 

The  distance  course  consists  presently  of  three  examinations  (with  five  assignments  each)  which  each 
have  to  be  completed  within  a single  week.  The  course  is  concluded  by  a 24-hour  final  examination 
consisting  of  eight  assignments,  which  determines  the  final  grade  of  the  course  and  includes  at  least  two 
assignments  from  each  examination  category.  Each  examination  (including  the  final)  is  issued  three  times  per 
course.  To  encourage  as  many  students  as  possible  to  aim  for  the  top  grade,  the  workload  of  the  students  is 
maximized  during  the  start,  thereby  giving  them  sufficient  time  at  the  end  of  the  course  to  perform  studies  at 
a more  advanced  level  than  is  required  to  pass  the  initial  examinations. 

The  basic  idea  behind  the  design  of  the  VMA  system  was  to  break  down  common  problems  within  the 
area  of  discrete  mathematics  into  basic  elements  and  then  recombine  such  elements  into  a more  complex  final 
result  (assignments).  Each  area  showed  however  to  be  fundamentally  different  from  the  others  and  many 
problems  were  in  relative  terms  resolved  by  ad  hoc  solutions.  The  VMA  system  consists  currently  of  16 
assignment  generation  engines,  see  Table  1. 


Table  1.  A list  of  the  VMA  assignment  generation  engines 


Engine 

Examination 

Category 

Models 

Truth  Table 

1 

Logic 

8 

Propositional  Logic 

1 

Logic 

16 

Predicate  Logic 

1 

Logic 

102 

Karnaugh  Diagram 

1 

Logic 

2 

Venn  Diagram 

1 

Set  Theory 

2 

Set  Equation 

1 

Set  Theory 

12 

Radix 

2 

Numeral  Systems 

6 

RSA  Encryption 

2 

Modular  Systems 

1 

Diophantine  Equation 

2 

Modular  Systems 

1 

Matrix  Multiplication 

2 

Arrays 

2 

Proof  by  Induction 

2 

Induction 

2 

Minimum  Spanning  Tree 

3 

Graph  Theory 

4 

Combinatorics  - Level  1 

3 

Combinatorics 

10 

Combinatorics  - Level  2 

3 

Combinatorics 

11 

Combinatorics  - Level  3 

3 

Combinatorics 

9 

Combinatorics  - Level  4 

3 

Combinatorics 

14 

In  its  simplest  form,  an  engine  or  a model  could  in  theory  consist  of  a number  generator  and  an 
environmental  context  generator.  The  number  generator  is  here  defined  as  the  generator  of  the  calculations 
involved  in  solving  a problem.  The  environmental  context  generator  is  the  context  in  which  a problem  can 
occur,  typically  expressed  by  words.  By  the  multiplication  of  the  numerical  set  with  the  environmental,  a 
matrix  of  possible  assignments  can  be  created.  Although  this  is  a straightforward  idea  it  showed  to  be 
difficult  to  implement  across  the  board  due  to  the  great  differences  between  the  types  of  problems  that  had  to 
be  formulated  and  solved.  Of  all  16  engines  the  simplest  to  implement  was  the  matrix  multiplication  engine 
(that  was  included  in  the  course  to  support  students  who  had  not  previously  studied  linear  algebra),  which 
only  showed  to  require  two  generation  models.  This  assignment  was  also  by  far  the  simplest  to  solve  for  the 
students.  In  such  assignment,  two  matrices  are  allocated  and  randomly  filled  with  small  integers,  such  that  a 
predefined  number  of  elements  are  non-zeros  to  regulate  the  difficulty  of  the  assignment.  Matrix  size  is  made 
to  vary  in  addition  to  the  context  in  which  the  multiplications  are  performed. 

The  most  difficult  engine  to  develop  was  for  predicate  logic,  which  showed  to  require  about  a hundred 
models.  This  assignment  type  was  the  only  one  in  the  course  that  was  transformed  into  a multiple-choice 
section  (three  questions  per  assignment  with  eight  choices  per  question).  Traditionally  in  this  course  the 
students  were  asked  to  express  formal  language  as  predicate  logic.  The  backside  for  the  teacher  was  that  it 
could  take  in  average  15  minutes  to  analyze  the  solution  for  each  student,  to  grade  it  and  write  an  appropriate 
reply.  By  making  this  a multiple  choice  assignment  and  instead  focus  the  effort  to  provide  for  very  detailed 
solutions  for  the  students  to  study,  the  average  grading  and  feedback  time  for  this  assignment  was  for  each 
student  reduced  to  below  30  seconds  and  feedback  from  the  students  on  the  received  grade  was  nearly 
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eliminated.  In  addition,  since  the  generation  of  combinatorics  problems  was  difficult  to  automate,  four 
engines  were  devised,  each  designed  to  generate  assignments  with  a fixed  level  of  difficulty. 

One  of  the  main  challenges  in  designing  a system  like  VMA  showed  to  be  balancing.  The  level  of 
difficulty  for  each  assignment  is  for  instance  recommended  to  reside  within  predefined  limits,  else  the 
variation  between  the  examinations  could  occasionally  turn  out  to  be  out  of  proportion.  Some  natural 
variation  is  expected  even  in  manual  development  of  examinations,  but  in  the  case  of  automatic  generation  of 
assignments  the  predictability  of  the  level  of  difficulty  for  each  individual  assignment  is  even  more  important 
for  unsupervised  generation.  As  an  example  the  VMA  Diophantine  equation  engine  is  able  to  generate 
potentially  a large  number  of  problems,  but  only  selects  those  with  moderately  complex  solutions,  thereby 
eliminating  equations  with  no  solution  or  only  simpler  solutions  and  equations  that  require  relatively 
extensive  calculations.  Such  trial  and  error  process  was  used  in  some  engines  to  ensure  that  proper  level  of 
difficulty  was  found,  but  only  if  it  was  not  possible  to  predict  the  complexity  of  the  generated  assignment  in 
advance  by  any  other  means.  Another  question  was  the  relative  balancing  between  the  difficulty  levels  of 
different  assignments.  A basic  inter-assignment  moderator  was  developed  for  compilation  of  examinations  so 
that  easier  assignments  would  counter  more  difficult  ones,  thereby  keeping  the  total  level  for  an  examination 
paper  relatively  constant. 

A system  was  in  addition  implemented  for  engines  incorporating  the  largest  number  of  models  to  ensure 
that  it  would  take  a few  course  occasions  before  the  same  model  was  used  twice.  The  main  reason  for  this 
was  that  one  of  the  balancing  factors  in  addition  was  the  distribution  of  assignments  with  respect  to  the  levels 
of  required  creativity  to  solve  a certain  problem.  The  less  creative  assignments  were  in  this  work  categorized 
as  “procedural”  (such  as  matrix  multiplication)  and  are  typically  based  on  a few  models  and  the  more  original 
as  “creative”  (such  as  the  higher  level  combinatorics  problems),  based  typically  on  multiple  models.  The 
overall  division  is  currently  considered  to  be  relatively  even  between  procedural  and  creative  elements 
throughout  the  whole  course.  As  a note,  procedural  problems  are  defined  as  the  ones  were  the  student  has  to 
carefully  follow  the  instructions  of  an  assignment  and  double-check  its  own  calculations  to  maximize  its 
score.  The  creative  problems  test  the  knowledge  of  the  students  on  a deeper  level  by  the  introduction  of 
problems  that  are  usually  not  included  in  precise  terms  in  the  textbook  or  can  be  found  on  the  Internet,  but 
that  easily  are  solved  if  the  essence  of  the  subject  is  thoroughly  understood.  As  a rule  the  generation  of 
procedural  problems  showed  to  be  the  most  straightforward  to  automate,  while  the  creative  problems  showed 
to  be  relatively  difficult  to  automate  in  the  sense  that  multiple  models  were  required  and  that  it  typically  took 
at  least  five  times  as  much  time  to  develop  an  engine  for  a creative  assignment. 

The  addition  of  uncommon  (but  plausible)  constraints  to  conventional  assignments  showed  in  addition  to 
be  a useful  tool  in  the  design  of  original  assignments.  Not  only  did  it  make  grading  in  some  occasions 
significantly  easier  in  procedural  assignments  by  making  all  solutions  to  look  more  or  less  alike  (and  almost 
identical  to  the  generated  solution  provided  by  the  VMA),  but  it  also  excluded  the  possibility  that  the  student 
could  have  used  an  automatic  generator  on  the  Internet  to  solve  the  problem  without  a deeper  understanding 
of  the  subject. 


3.  RESULTS 

Although  automatic  item  generation  could  be  considered  to  be  relatively  difficult  to  implement  within 
discrete  mathematics,  it  proved  to  be  possible  for  a single  programmer  spending  one  time  unit  in  the 
development  of  this  system  to  save  potentially  200  time  units  for  a traditional  mathematics  educator.  No 
decrease  was  further  detected  in  the  quality  of  the  generated  assignments  and  solutions  compared  to  manually 
developed  ones  during  previous  years  in  the  same  course.  This  assumption  is  partly  based  on  student 
evaluations,  which  consisted  of  50  replies  out  of  200  participating  students.  One  positive  side-effect  of  the 
application  of  this  new  system  was  that  since  it  was  developed  with  a possible  future  automatic  grading 
system  in  mind,  the  manual  grading  process  was  significantly  simplified  as  well.  This  was  appreciated  by  the 
students  since  grades  could  be  received  typically  within  24  hours  after  examination  deadline.  The  downside 
was  that  the  procedural  elements  in  the  course  could  only  be  solved  in  a very  specific  fashion  to  ensure  rapid 
grading,  and  since  all  students  received  identical  assignments,  this  made  it  difficult  to  detect  plagiarism.  The 
creative  assignments  did  not  suffer  from  this  flaw  since  they  could  be  solved  in  multiple  ways,  but  made  the 
grading  process  more  challenging. 
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4.  CONCLUSION  AND  FUTURE  WORK 

Considering  how  well  this  system  proved  to  work  in  practice  (as  expected  from  previous  projects  similar  to 
this),  there  is  really  no  reason  why  such  systems  should  not  in  the  future  become  a natural  integral  part  of 
mathematics-based  education  across  the  board.  Once  there  are  sufficiently  many  automatic  item  generators  to 
choose  between  for  a professor  so  that  it  takes  shorter  time  for  the  students  to  actually  learn  the  subject  in 
question  compared  with  using  similar  systems  to  automatically  find  a solution,  this  could  significantly  reduce 
the  required  time  for  the  generation  of  examinations.  Since  the  lifespan  of  VMA  is  likely  to  be  considerably 
shorter  than  2048  course  occasions,  such  system  could  in  practice  be  used  to  provide  each  student  with  an 
individual  assignment,  increasing  the  flexibility  by  which  examinations  are  issued  and  at  the  same  time 
decreasing  the  risk  for  plagiarism  in  both  campus  and  distance  courses.  While  most  VMA  engines  are  at  this 
stage  considered  to  be  complete,  for  sustained  quality  over  time,  the  recommended  number  of  generation 
models  for  the  combinatorics  engines  is  at  a minimum  24  models  per  engine  or  preferably  the  double.  In 
addition,  the  engine  for  proof  by  induction  is  recommended  to  be  extended  by  one  extra  model. 


ACKNOWLEDGEMENT 

Many  thanks  to  Dr.  Bo  Goran  Johansson  for  the  main  design  of  the  course  structure  as  defined  in  Table  1, 
which  was  of  great  help  during  the  design  of  the  VMA  system. 


REEERENCES 

Apple  Xcode,  Apple  Inc.,  10  February  2013  <https://developer.apple.com/xcode>. 

Beevers,  C.  & Jackson,  D.  H.,  2000.  Automatic  Assessment  - Mathwise  and  Beyond.  In  Proc  Twelfth  International 
Conference  on  Technology  and  Collegiate  Mathematics,  San  Francisco,  1999,  1/01/70,  pp.  19-23. 

Maple  TA,  Waterloo  Maple  Inc.,  20  January  2013  <http://maplesoft.com/products/mapleta>. 

NCES,  2005.  Online  Assessment  in  Mathematics  and  Writing:  Reports  From  the  NAEP  Technology-Based  Assessment 
Project,  Research  and  Development  Series.  National  Center  for  Education  Statistics,  Institute  of  Education  Sciences, 
U.S.  Department  of  Education,  1990  K Street  NW,  Washington,  DC  20006-5651. 

Pead,  D.,  2010.  On  Computer-Based  Assessment  of  Mathematics.  PhD  Thesis,  University  of  Nottingham. 

Rasila,  A.  et  al,  2010.  Automatic  Assessment  in  Engineering  Mathematics:  Evaluation  of  the  Impact.  In  ReflekTori  2010 
Symposium  of  Engineering  Education,  Aalto  University  School  of  Science  and  Technology,  pp.  37-45. 

TeXShop,  24  January  2013  <http://pages.uoregon.edu/koch/texshop>. 


368 


lADIS  International  Conference  e-Learning  2013 


DIRECTION  OF  CONTENTS  DEVELOPMENT  FOR 

SMART  EDUCATION 


YoungSun  Park,  SangJin  An  and  YoungJun  Lee 

Korea  National  University  of  Education 
Chungbuk,  Korea 


ABSTRACT 

The  aim  of  this  study  is  to  suggest  a direction  of  developing  SMART  education  contents  for  its  effective  implementation 
by  analyzing  the  status  of  educational  informatization  policies  in  Korea.  Korean  government  has  built  the  information 
and  communication  infrastructure,  and  provided  teachers  and  students  with  various  kinds  of  contents.  And,  in  preparation 
of  the  SMART  education  which  will  be  implemented  in  all  elementary,  middle,  and  high  schools  in  2015,  the  government 
plans  to  develop  digital  textbooks  and  supply  contents  of  cyber  home  learning  system,  Edunet,  and  private  companies  by 
linking  digital  textbooks.  In  order  to  realize  the  merits  of  the  SMART  education  to  the  maximum,  it  is  necessary  to 
develop  practical  and  participatory  contents  per  level  of  student  abilities  and  suggest  them  to  students. 


KEYWORDS 
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1.  INTRODUCTION 

Development  of  information  and  communications  technology  and  the  popularization  of  smart  devices  have 
given  rise  to  huge  changes  in  our  life  styles  across  social,  economic  and  cultural  areas.  Such  changes  also 
affected  the  education  area.  It  is  common  to  see  students  or  office  workers  listen  to  mobile  lectures  using 
smart  phone  in  bus  or  subway.  In  response  to  such  social  and  educational  changes,  the  Korean  government 
issued  the  SMART  education  policy  in  2011.  The  SMART  education  is  not  simply  learning  utilizing  the 
smart  infrastructure.  It  means  to  provide  systematic  learning  service  to  learners,  based  on  the  close 
connection  of  the  smart  environment  and  learning  principles.  And,  to  make  such  service  effective,  it  is 
necessary  to  provide  excellent  learning  contents  with  the  foundation  of  the  smart  infrastructure.  Thus,  in  this 
study,  by  analyzing  the  status  of  educational  informatization  policies  and  characteristics  of  the  SMART 
education,  discusses  the  direction  of  contents  development  for  SMART  education. 


2.  EDUCATIONAL  INFORMATIZATION  POLICIES 

Korean  government  has  promoted  various  policies  through  the  educational  informatization  project.  The 
educational  informatization  policies  have  developed  with  the  evolution  of  the  Internet  environment  and  with 
various  educational  purposes.  Currently  in  Korea,  there  are  various  educational  informatization  policies, 
which  provide  contents  for  teaching  and  learning  (Jeong,  2012).  Let's  examine  some  typical  policies. 

2.1  Cyber  Home  Learning  System 

Cyber  Home  Learning  System  is  the  learning  service  the  government  supports  to  help  students  compensate 
for  what  they  lack  using  the  Internet  learning  contents  which  is  designed  to  reduce  high  private  education 
cost  and  narrow  the  educational  gap.  Cyber  Home  Learning  System  began  to  be  executed  in  2005.  Students 
who  want  to  study  through  Cyber  Home  Learning  System  can  use  the  program  free  of  charge  after  registering 
their  names  on  Cyber  Home  Learning  System  homepage.  The  contents  of  Cyber  Home  Learning  System  are 
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developed  based  on  national  level  curriculum.  The  status  of  the  contents  is  shown  in  Table  l(Lim,  2012). 
Contents  are  different  per  level,  and  each  student  can  choose  the  level  contents  of  which  he  or  she  thinks  are 
proper  to  his  or  her  ability.  However,  there  has  been  criticism  that  the  contents  simply  repeat  the  textbooks  or 
they  are  just  description  of  general  matters  without  giving  concrete  examples  (Kim,  2011). 


Table  1.  The  status  of  Cyber  Home  Learning  System  contents  (Lim,  2012) 


Classification 

Contents 

Subject 

The  grades:  Korean,  Society,  Mathematics,  Science,  English 

Type 

Self-study  type  contents  produced  with  the  Flash  program 

Composition 

Explanation  of  learning  contents.  Questions  and  explanations 

2.2  Edunet 

Edunet  is  the  central  teaching  and  learning  center  established  in  2004.  It  is  established  to  research  and 
support  teaching  and  learning.  It  provides  students  with  self-study  contents,  and  teachers  with  various 
multimedia  materials  for  teaching.  As  of  2012,  Edunet  membership  is  6.36  million  consisting  of  teachers, 
students  and  parents.  It  keeps  acquiring  education  contents  from  the  department  of  education,  public 
organizations  and  private  companies  (Lim,  2012).  It  has  been  found  that  students  use  Edunet  mainly  for  self- 
study  (50.8%)  or  for  doing  homework  (45.7%).  And,  61.1%  of  teachers  use  it  to  prepare  for  teaching. 
However,  some  problems  of  Edunet  which  need  to  be  remedied  have  been  pointed  out,  such  as  too  complex 
make-up  of  the  menu  and  the  difficulty  to  grasp  what  kinds  of  contents  are  supplied  (Kim,  2012).  The  status 
of  Edunet  contents  is  shown  in  Table  2. 


Table  2.  The  status  of  Edunet  contents  (http://www.edunet.net) 


Classification 

Teacher 

Student 

Subject 

The  P‘-9"’ grades:  all  subjects 

The  10'*'  grade:  Korean,  Society,  Mathematics, 

Science,  English 

The  4‘*'-9'*’  grades:  all  subjects 

The  lO*  grade:  Korean,  Society,  Mathematics, 

Science,  English,  and  Essay  writing 

Type 

Multimedia  materials  for  teaching  such  as  video, 
animation,  and  flash,  etc. 

Materials  for  self-study  such  as  flash,  video,  and 
animation,  etc. 

Composition 

Proposition  on  teaching  and  learning  process. 
Teaching  materials  per  level,  Materials  for 
evaluation 

Different  materials  per  level 
(basics/supplement/deepening) 
Explanation  of  learning  contents 

2.3  Digital  Textbook  Project 

In  2007,  the  Ministry  of  Education,  Science  and  Technology  announced  the  adoption  of  Digital  Textbooks, 
and  developed  the  prototypes  of  those  textbooks,  to  support  individualized  learning  activities,  and  respond  to 
the  changed  characteristics  of  learners  who  are  digital  natives.  Digital  Textbooks  contain  not  only  contents  of 
book-type  textbooks,  but  various  multimedia  materials,  dictionaries,  and  evaluation  questions,  and  can  be 
used  at  home  and  school  without  the  constraints  of  time  and  space(Ministry  of  Education,  Science  and 
Technology,  2007).  Digital  Textbooks  were  tested  in  20  schools  in  2008,  112  in  2009,  132  in  2010,  63  in 
2011,  and  46  in  2012.  The  schools  which  test  runs  those  Digital  Textbooks  were  provided  with  the 
infrastructure  such  as  tablet  PC,  electronic  blackboard,  wireless  Internet,  and  remote  control  program  (Lim, 
2012).  Researches  on  the  effects  of  Digital  Textbooks  have  been  done  continuously  through  the  operations  of 
schools  adopting  Digital  Textbooks.  It  has  been  found  that  those  textbooks  contribute  to  the  improvement  of 
various  capacities  of  students  such  as  academic  achievement,  learning  flow,  self-directed  learning  ability,  and 
problem-solving  ability  (Seo  et  ak,  2009;  Noh,  2011;  Byun  et  ak,  2011). 

Currently,  Digital  Textbooks  use  the  Window/Linux  comprehensive  platform,  and  the  platform  and 
contents  of  it  can  be  downloaded  through  the  web.  The  status  of  Digital  Textbook  contents  is  shown  in  Table 
3.  As  shown  in  the  table,  typical  characteristics  of  Digital  Textbooks  are  that  they  provide  English  contents 
per  level  to  support  individualized  learning  in  classroom.  Per -level  English  contents  provide  different 
contents,  activities,  and  exercises  for  students  in  different  levels  for  every  unit  of  the  textbook,  which  has  not 
been  possible  by  traditional  textbooks.  On  the  contrary,  some  contents  like  simulations  have  linear  formats 
which  students  simply  click  or  drag  them  and  confirm  the  results,  failing  to  reflect  various  situations  where 
students  can  meet  in  their  learning  process  and  restricting  active  thinking  of  students  (Kim,  2012). 
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Table  3.  The  status  of  Digital  Textbook  contents  (http://webhard.edunet.net) 


Classification 

Contents 

Subject 

The  4*-6*  Grades:  Korean,  Society,  Mathematics,  Science 
The  3'‘*-6*  Grades:  English  per  level 
The  7*  grade:  English,  Science 

Type 

Learning  assistant  materials  such  as  flash,  video,  animation,  and  photos,  etc. 

Composition 

Complementary  explanation  on  learning  contents,  game,  drawing,  mindmap,  simulation, 
virtual  museum,  and  evaluation  questions,  etc. 

2.4  Common  Issues  of  Digital  Contents 

There  is  not  much  difference  in  the  degree  of  difficulty  of  per -level  contents,  and  not  enough  consideration  is 
given  to  students  whose  levels  are  very  low,  or  very  high.  Furthermore,  most  of  contents  are  designed  as 
teacher-centered  one-way  format  where  learner  can  only  receive  learning  contents  passively.  Such 
characteristics,  when  combined  with  cyber  learning,  can  discourage  students,  lowering  their  interests  and 
concentrations  in  learning.  Another  problem  is  that,  since  all  the  contents  provided  through  Cyber  Home 
Learning  System,  Edunet,  and  Digital  Textbooks  are  web-based,  they  cannot  be  used  in  various  devices  like 
smart  phone  or  smart  pad. 


3.  SMART  EDUCATION 


3.1  Definition  and  Characteristics 

In  2011,  the  Ministry  of  Education,  Science  and  Technology  announced  SMART  education  development 
strategy  as  ways  to  respond  to  the  changing  educational  environment  due  to  the  evolution  of  Internet 
environment  and  popularization  of  smart  devices  and  to  strengthen  learner  capabilities  in  the  21st  century. 
SMART  education  is  to  embody  the  individualized  learning  system  by  actively  applying  information  and 
communication  technology  and  network  resources,  and  changing  education  contents,  methods,  evaluation, 
and  environment.  To  realize  the  goal,  the  government  established  a detailed  development  plan  on  a wide 
range  of  issues  such  as  development  and  application  of  Digital  Textbooks,  activation  of  online  teaching  and 
evaluation,  building  of  sharing  environment  to  utilize  of  convenient  and  safe  contents,  and  building  of  cloud 
education  service.  It  announced  its  plan  to  implement  SMART  education  across  the  nation  in  2015.  The 
characteristics  of  the  SMART  education  can  be  classified  into  five  categories  which  start  with  the  letters,  S, 
M,  A,  R,  and  T,  as  displayed  in  Table  4(Ministry  of  Education,  Science  and  Technology,  201 1). 


Table  4.  Characteristics  of  SMART  education  (Ministry  of  Education,  Science  and  Technology,  2011) 


Category 

Characteristics 

Self-directed 

Learner-centered  education,  teacher  plays  the  role  of  mentor; 

Provision  of  the  learning  management  system  where  the  learner  can  plan,  manage,  and  operate 
one's  own  learning 

Motivated 

Emphasis  on  experience-oriented,  problem  solving-oriented  educational  methods  to  make  the 
learner  learn  with  interest 

Adaptive 

Realization  of  custom-made  learning  suited  to  different  levels  and  aptitudes  of  learners 

Resource  Free 

Letting  learners  use  various  contents  freely  through  the  open  market,  the  distrihution  system  of 
contents; 

Enlargement  of  sharing  of  learning  domestic  and  foreign  resources,  and  of  cooperative  learning 
through  social  networking 

Technology 

Embedded 

Provision  of  the  environment  where  the  learner  can  learn  at  anyplace  anytime  through  information 
technology  like  smart  phone,  tablet  PC,  cloud  computing,  and  4G  network 
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3.2  Planned  Contents  Provision 


In  SMART  education,  contents  are  provided  through  digital  textbooks  and  open  market.  Digital  Textbooks 
have  been  developed  and  applied  in  pilot  schools  from  2007.  They  are  designed  to  be  used  through  PC,  so 
they  cannot  be  used  in  various  devices  like  smart  phone  or  smart  pad.  So,  considering  smart  educational 
environment.  Digital  Textbooks  which  will  be  developed  in  the  future  should  be  developed  to  be  used  in 
various  devices  like  PC,  smart  pad,  and  smart  TV. 

The  learning  management  system  is  run  for  the  effective  management  and  use  of  Digital  Textbook 
contents.  It  is  expected  to  build  the  contents  sharing  environment  by  linking  contents  supplied  by  Edunet,  and 
Cyber  Home  Learning  System,  etc.  with  open  market.  Such  contents  will  be  used  not  only  for  teaching  in 
class  and  individual  learning  at  home,  but  for  online  class  which  is  part  of  regular  class.  And,  to  respond 
rapidly  to  the  changes  of  educational  environment  following  the  textbook  publication  system  to  government 
approved  textbooks  and  revision  of  curricula.  Digital  Textbook  contents  also  will  change  from  the  system 
where  the  government  provides  all  the  contents  to  the  one  various  sources  share  the  responsibility  of 
providing  contents.  Accordingly,  the  contents  part  of  Digital  Textbooks  will  be  taken  charge  of  by  textbook 
publishing  companies,  the  part  of  viewer  and  SMART  education  platform  will  be  done  by  the  government, 
and  development  and  provision  of  contents  through  the  link  with  the  open  market  will  be  done  by  both  the 
government  and  private  companies.  Figure  1 shows  the  concept  of  digital  textbooks  (Ministry  of  Education, 
Science  and  Technology,  2011;  Lim,  2012). 
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Figure  1.  Digital  Textbook  concept  map  (Lim,  2012) 


3.3  The  Direction  of  Contents  Development  for  SMART  Education 

The  following  are  directions  of  contents  development  for  SMART  education  based  on  the  analysis  of  the 
status  of  educational  informatization  policies  and  characteristics  of  SMART  education. 

First,  contents  should  be  developed  per  level.  To  make  SMART  education  an  individualized  one,  it  is 
necessary  that  Digital  Textbooks  which  will  be  developed  in  the  future  should  provide  per -level  contents  in 
learning  units  of  all  subjects  where  degree  of  difference  is  differentiated.  In  addition,  in  order  not  to  make 
contents  of  Cyber  Home  Learning  System  simple  repetition  of  textbook  contents,  but  to  help  the  learner 
understand  the  contents,  it  should  be  revised  to  add  various  examples  and  reference  materials. 

Second,  in  the  aspect  of  contents  design,  contents  should  be  practical  and  learner-participating  ones. 
Contents  should  not  be  of  the  one-way  format  where  the  learner  checks  the  results  by  simply  clicking  and 
dragging.  They  should  be  practical  ones  reflecting  the  various  restrictive  conditions  which  can  take  place  in 
real  learning.  Furthermore,  as  the  SMART  education  aims  at  active  participation  of  learners  like  their 
experiences  and  problem-solving  processes,  it  should  provide  a variety  of  experiential  learning  contents  like 
simulation,  virtual  reality,  augmented  reality,  and  educational  game.  Such  contents  make  it  possible  for 
learners  to  experience  various  types  of  learning  which  they  have  not  been  able  to  acquire  in  classroom 
because  of  the  constraints  of  time  and  space. 
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Third,  in  the  aspects  of  contents  production  and  supply,  contents  should  be  produced  observing  contents 
development  standards,  and  convenience  to  access  the  contents  of  users  should  be  considered.  Since  digital 
textbooks  are  developed  by  various  publishing  companies,  it  is  necessary  for  the  government  to  suggest 
contents  development  standards.  Not  only  Digital  Textbooks  but  existing  Cyber  Home  Learning  System  and 
Edunet  contents  need  to  be  redeveloped  observing  contents  development  standards  in  order  to  make  them  be 
used  on  various  devices.  And,  since  various  contents  will  be  provided  through  open  market,  it  is  necessary  to 
build  database  in  order  to  let  users  more  convenient  search  for  and  download  materials. 


4.  CONCLUSION 

Currently,  in  Korea,  various  educational  informatization  policies  are  implemented.  Up  to  now,  those  policies 
have  achieved  quantitative  growth  in  terms  of  building  the  information  communication  infrastructure  and 
supplying  various  contents  to  learners.  And,  now  the  government  wants  to  upgrade  effectiveness  of  education 
through  the  SMART  education  based  on  more  advanced  information  technology.  For  successful 
implementation  of  the  SMART  education,  core  elements  of  teaching  and  learning  should  be  considered,  first 
of  all.  In  this  aspect,  development  of  good-quality  contents  is  the  most  important  aspect.  When  orientations 
of  the  SMART  education  and  digital  contents  meet  together,  SMART  education  will  be  realized  for  learners. 
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ABSTRACT 

On-line  training  is  becoming  an  interesting  phenomenon  in  Australia  and  has  attracted  a lot  of  interest  across  many 
industries  and  businesses  (Chan  and  Ngai,  2007).  The  research  reported  here  looks  at  the  use  of  online  training  in 
corporations  in  Australia.  It  focuses  on  two  aspects  of  online  training,  the  factors  that  ‘warrant’  its  success  as  well  as  on 
benefits  of  being  involved  in  online  training. 

This  exploratory  study  into  the  issues  prevalent  to  organisations  in  Australia  was  accomplished  through  a survey  of  top 
500  Australian  companies. 

The  findings  from  this  research  shows  that  a lot  of  success  factors  of  online  training  identified  in  the  literature  have  been 
identified  in  the  Australian  environment  as  well.  These  include:  financial  support,  cross-functional  team,  alignment  of  the 
software  and  organisational  standards  (Driscoll,  1998);  top  management  support,  skilled  work  force  (Talib  et  al,  2010); 
just  in  time  training  (Chan  and  Ngai,  2007);  and  usability  and  transferability  (Wang  et  al,  2004). 

Additionally,  the  findings  also  confirm  the  findings  from  the  literature  about  major  benefits  of  online  training.  These 
include:  cost  effectiveness  (Wang  et  al,  2004);  efficiency,  competitive  advantage,  receiving  training  any  time,  any  place, 
ability  to  provide  custom  tailored  solutions  (Chan  and  Ngai,  2007),  24  hours  accessibility,  flexibility,  relearning  skills; 
and  improving  knowledge  and  practice  of  preventive  oral  health  (Talib  et  al,  2010). 


KEYWORDS 

Online  training,  Australia 


1.  INTRODUCTION 

The  online  training  has  been  “defined”  by  few  authors  as  activities  that  uses  Internet,  intranet,  www,  for 
learning  purposes  (Hall,  1997);  a way  of  communication  that  uses  information  technology  in  order  to  provide 
instructions  24/7  and  is  geographically  independent  (Chamers  and  Lee,  2004);  (Davis,  2001).  Some  authors 
see  the  advent  of  the  Internet  as  a vehicle  for  delivering  this  important  service  (Greenberg,  1998). 

Online  training  allows  collaborative  learning  environment,  delivering  the  training  in  a more  cost-effective 
and  efficient  way,  as  well  as  without  the  need  to  attend  the  classes  (Levin  et  al,  1999),  and  (Davis,  2001). 

Some  of  the  reasons  for  the  companies  to  get  engaged  in  this  activity  include  its  flexibility,  adaptability, 
low  cost,  and  just-in-time  training  (Chamers  and  Lee,  2004);  (Simmons,  2002). 


2.  LITERATURE  REVIEW 

Online  training  is  becoming  more  and  more  popular  in  recent  years  and  because  of  its  ability  to  provide  a 
variety  of  benefits  has  been  adopted  in  many  organisations  worldwide.  In  the  last  few  years  it  has  made  a 
profound  (Chamers  and  Lee,  2004).  According  to  Sloman  ( 2001),  it  is  attractive  to  a large  number 
companies  and  businesses  all  over  the  world. 

In  addition,  according  to  Talib  et  al  (2010)  the  use  of  online  training  has  witnessed  the  increase  in 
knowledge,  efficiency,  etc.  in  many  industries.  According  to  the  same  author,  it  also  supports  the  practice  of 
preventive  oral  care,  in  the  health  industry.  Furthermore,  the  online  training  can  be  of  great  use  in 
telerehabilitation  by  providing  an  alternative  to  the  classical  clinic-based  approach  (Burdea  et  al,  2000). 
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To  be  successful  in  any  field  and  to  be  able  to  achieve  benefits  the  businesses  need  to  identify  those  few 
factors  that  can  “make  it  or  break  it”.  The  identification  of  these  success  factors  is  one  of  the  most  important 
tasks  for  the  businesses  that  want  to  be  involved  in  online  training. 

One  of  the  best  definitions  of  success  factors  is  provided  by  Laudon  and  Laudon  (1988;  1998),  and  has 
been  cited  by  Hossain  (1999):  “success  factors  are  small  numbers  of  easily  identifiable  operational  goals 
shaped  by  the  industry,  the  firm,  the  manager,  and  the  environment  that  assure  the  success  of  an 
organization”. 

In  the  context  of  online  training,  and  for  that  matter  in  any  other  context,  the  achievement  of  benefits  is 
only  possible  through  the  identification  of  success  factors.  Some  of  the  major  success  factor  identified  by  the 
literature  are:  top  management  support,  positive  attitude  towards  online  training  adoption,  innovativeness, 
knowledge  about  IT,  user-friendliness,  and  championing  the  online  training  in  an  organisation  financial 
support;  cross  functional  team;  conformance  to  the  corporate  standards;  matching  the  technology  and  the 
training  topic;  right  complexity  of  the  content;  familiarity  with  the  tools;  ability  to  provide  continuous 
improvement,  skilled  work  force,  ability  to  provide  flexible  services  (Aguinis  et  al,  2009);  (Driscoll,  1998); 
(Chan  and  Ngai,  2007);  (Chamers  and  Lee,  2004);  (Simmons,  2002);  (McGinnis  and  Ackelsberg,  1983); 
(Quinn,  1985);  (Westmoreland  et  al,  2010)  and  (Thong  and  Yap,  1995). 

Among  the  main  reasons  for  many  companies,  regardless  of  their  size,  to  participate  in  any  business 
endeavor  is  to  be  successful  and  to  achieve  benefits  from  it.  Online  training  is  not  any  different  in  that 
respect.  It  has  provided  organisations  with  opportunities  to  achieve  benefits  by  reducing  operation  costs, 
improving  customer  service,  and  gaining  business  efficiencies  (Chan  and  Ngai,  2007);  (Markham  et  al, 
2009);  (Gasparetti  et  al  2009);  (Carpenter  et  al,  2010);  and  (Kleinpell  et  al,  201 1). 

Additionally,  online  training  can  help  organisation  in  providing  custom  tailored  training,  achieving  time 
savings,  reusing  of  programs/trainings  as  many  times  as  they  need  it,  allowing  24/7  delivery,  and  information 
sharing  (Chan  and  Ngai,  2007).  Online  training  is  about  building  better  relationships  between  its  participants 
and  allowing  its  consumers  to  search  for  products  and  check  for  customer  service  information  online,  helping 
in  decision  making  process  (Raupach  et  al,  2009);  (Lockhart  and  Smith,  2009);  (Thistlethaaite,  et  al,  2007); 
and  (Wang  et  al,  201 1). 

Achieving  benefits  in  this  field,  is  however,  quite  difficult,  and  to  make  sure  they  are  successful, 
organizations  must  really  endeavor  in  identifying  factors  (CSF)  that  can  bring  those  benefits  around. 


3.  RESEARCH  METHODOLOGY 

The  research  represented  in  this  paper  is  a part  of  a larger  research  into  online  training  accomplished  through 
a survey  of  top  500  organisations  in  Australia.  It  was  guided  by  the  following  research  questions: 

a)  What  are  the  Critical  Successful  Factors  of  online  training  in  top  500  Australian  organisations? 

b)  Is  there  any  difference  among  identified  critical  success  factors  in  Recreation  and  other  services,  and 
finance/banking  industry? 

The  survey  questionnaire  was  piloted  and  subsequently  sent  to  the  top  500  organisations  in  Australia.  The 
participants  were  asked  to  list  all  the  critical  success  factors  of  online  training  to  their  organisations,  and  rank 
the  importance  of  each  of  the  factors.  The  ranking  included  using  0 to  5 Likert  scale,  0 being  the  success 
factor  not  identified,  1 being  list  important  and  5 being  most  important  factor  identified. 

The  responses  collected  were  analysed  using  the  non-parametric  statistical  test/analyses  because  of  the 
non-parametric  nature  of  the  data  from  the  Likert  scale.  The  data  collected  for  the  research  reported  in  this 
paper  included  the  Rank  Analysis,  and  Kolmogorov-Smirnov  test. 

The  Rank  Analysis  was  conducted  to  find  out  what  are  the  most  important  critical  success  factors  in 
participating  organisations. 

The  Kolmogorov-Smirnov  test  was  conducted  to  find  out  if  there  were  significant  differences  among  the 
participating  organisations  from  finance/banking,  and  recreation  and  other  services  industry. 
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4.  FINDINGS  AND  DISCUSSIONS 

Out  of  500  questionnaires  sent  out  we  received  52  useable  responses,  making  the  response  rate  just  above 
10%.  The  Communications  industry  returned  the  highest  number  of  responses  - 11  in  total.  Insurance, 
Finance/Banking,  and  Wholesale/Retail  industries  returned  10  responses  each.  It  was  followed  by  the 
Recreation  and  other  services,  returning  9,  Education/Training  industry  1,  and  Community  services  1 
response. 

In  order  to  find  out  the  importance  of  each  CSF  in  the  whole  population  and  in  the  Finance/Banking,  and 
the  Recreation  and  other  services,  we  conducted  rank  analysis.  Table  1 shows  the  ranks  of  CSF  for  the  whole 
population  and  for  the  two  industries. 


Table  1.  The  rank  order  of  CSF  identified  in  the  whole  population,  finance/banking  and  Recreation  and  other  services 

industry 


Rank 

Whole  population 

Finance/banking  industry 

Recreation 

1 

Content 

Content 

Innovativeness 

2 

Skilled  work  force 

IT  knowledge 

Corporate  strategy 

3 

Top  management  support 

Cross-functional  team 

Content 

4 

Innovativeness 

Flexibility 

Top  management  support 

5 

Flexibility 

User  friendliness 

Skilled  work  force 

6 

Cross-functional  team 

Corporate  strategy 

Familiar  tools 

7 

User  friendliness 

Financial  support 

User  friendliness 

8 

Financial  support 

Topic  and  technology 
matching 

Financial  support 

9 

Familiar  tools 

Skilled  work  force 

Topic  and  technology  matching 

10 

Topic  and  technology  matching 

Top  management  support 

7 days  24  hours  availability 

11 

7 days  24  hours  availability 

Online  training  adoption 
readiness 

Cross-functional  team 

12 

Corporate  strategy 

Familiar  tools 

Online  training  adoption 
readiness 

13 

IT  knowledge 

Innovativeness 

IT  knowledge 

14 

Online  training  champion 

7 days  24  hours  availability 

Online  training  champion 

15 

Online  training  adoption 
readiness 

Online  training  champion 

Flexibility 

Table  1 show  that  9 critical  success  factors  of  online  training  are  represented  in  the  top  5 CSF  in  the 
whole  population,  finance/banking,  and  recreation  and  other  services  industry.  It  indicates  that  these  factors 
are  evenly  spread  throughout  all  3 populations.  It  also  indicates  that  the  recreation  and  other  services  industry 
has  higher  degree  of  “agreement”  with  the  whole  population  about  the  importance  of  these  top  critical 
success  factors  than  the  finance  and  banking  industry,  as  it  has  identified  4 out  of  5 factors  as  very  important 
from  their  organisations’  point  of  view.  The  9 identified  factors  represented  in  top  5 across  the  whole 
population  and  the  two  industries  include:  Content,  skilled  workforce,  top  management  support, 
innovativeness,  flexibility,  user-friendliness,  IT  knowledge.  Cross-functional  team,  and  Corporate  strategy. 

In  order  to  find  differences,  if  any,  among  identified  CSF  in  finance/banking  and  recreation  and  other 
services  industry  we  conducted  Kolmogorov-Smirnov  test.  The  results  of  Kolmogorov-Smirnov  test  (P>.05) 
shows  that  there  was  no  significant  difference  in  the  identified  success  factors  between  the  two  industries.  It 
indicates  that  both  industries  have  perceived  the  importance  of  the  identified  success  factors  similarly. 
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5.  CONCLUSIONS 

This  research  has  confirmed  that  the  critical  success  factors  of  online  training  that  have  been  found  in  the 
literature  have  also  been  found  in  the  top  500  Australian  companies.  The  two  industries,  focus  of  this 
research,  have  ranked  the  identified  critical  success  factors  similarly. 

The  research  also  indicates  that  top  ranked  success  factors  are  evenly  spread  throughout  the  whole 
population,  as  well  as  in  finance/banking  and  recreation  and  other  services  industry.  The  most  important 
success  factors  identified  in  the  whole  population  as  well  as  in  the  two  industries  include:  Content,  skilled 
workforce,  top  management  support,  innovativeness,  flexibility,  user-friendliness,  IT  knowledge.  Cross- 
functional team,  and  Corporate  strategy. 


6.  LIMITATIONS  AND  FURTHER  RESEARCH 

A major  limitation  of  this  research,  and  for  that  matter  all  the  survey  oriented  researches,  is  we  couldn’t 
clarify  some  of  the  issues  that  needed  some  clarification,  and  clear  the  doubts  raised  by  some  of  the  answers. 
A chance  to  clarify  these  and  some  other  issues  will,  hopefully,  be  presented  to  us  when  we  conduct  few  case 
studies  in  the  future  research  on  this  topic. 

In  the  near  future  we  will  attempt  to  conduct  few  case  studies  to  try  to  get  a better  understanding  of  the 
issues  we  have  our  doubts  after  this  survey.  Hopefully  we  would  be  able,  and  that  is  our  aim,  to  get  5 in  depth 
case  studies. 

After  completing  the  in  depth  case  studies,  we  would  endeavor  to  expend  our  research  to  large 
corporations  in  some  of  the  European  Union  countries,  to  get  the  sense  of  the  state  of  online  training  in  the 
Europe,  and  to  make  comparisons  with  the  Australia’s  large  corporations. 
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USING  FACEBOOK  AS  A VIRTUAL  CLASSROOM 
IN  A PUBLIC  UNIVERSITY  IN  MEXICO  CITY 
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ABSTRACT 

Since  Information  and  Communication  Technologies  have  been  developed,  many  changes  have  taken  place  in  society. 
Social  Networks  certainly  have  changed  communication  habits,  especially  among  young  people.  Nowadays,  Social 
Networks  are  used  as  a communication  system  every  day.  In  most  countries,  university  students  use  this  communication 
and  interaction  media  to  share  information,  images,  videos,  music,  etc.  but  essentially,  they  use  Social  Network  to 
socialize.  In  Mexico,  probably  more  than  90%  of  the  students  between  17  and  25  years  old  have  their  own  Facebook 
space.  In  this  way,  almost  every  student  in  the  university  knows  how  to  use  the  social  network,  especially  Facebook, 
nevertheless,  none  of  them  knows  how  to  use  virtual  classrooms  like  Moodle,  for  example.  This  is  a big  problem  for 
universities  which  are  interested  in  using  virtual  classrooms.  That  is  the  case  of  Metropolitan  Autonomous  University,  in 
Mexico  City  that  is  interested  in  using  Moodle  to  support  academic  activities. 

That  is  why  we  decided  to  change  the  paradigm:  if  we  can’t  take  students  to  virtual  classroom,  we  can  take  classroom  to 
students.  In  this  way  we  empowered  Facebook  as  a virtual  classroom. 

In  this  paper  we  describe  our  experience  in  using  Facebook,  no  just  as  a communication  media,  but  as  a real  virtual 
classroom,  in  a public  university  in  Mexico  City. 
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Social  Networks,  e-learning,  b-learning,  educative  Facebook. 


1.  INTRODUCTION 

In  July  2012,  a survey  was  conducted  at  Campus  Azcapotzalco  of  the  Metropolitan  Autonomous  University. 
A random  sample  of  470  students  was  selected.  The  data  collection  was  carried  out  during  ten  days,  from 
July  9 to  July  19,  2012.  The  survey  was  directly  applied  in  each  classroom.  The  target  of  the  study  was  to 
identify  the  technological  tools  which  are  more  used  by  students  in  their  activities  such  as  searching  and 
exchange  information,  generate  agreements,  sharing  tasks  and  interact  and  socialize  with  other  students. 
Later,  we  could  use  those  results  to  design  and  release  didactic  strategies  for  promoting  the  learning. 

The  results  pointed  out  among  any  other  things,  more  than  90%  of  students  usually  used  the  Social 
Networks  every  day.  Nowadays  more  than  90%  of  the  Metropolitan  Autonomous  University  students  use 
Facebook  for  socializing  with  their  classmates.  This  average  corresponds  surprisingly  with  other  studies 
(Chuck  Martin  (2009),  Connie  Varnhagen  (2011),  Nicola  Cavalli,(201 1),  Marisol  Gomez,  (2011))  . It  is 
interesting  that  while  in  2006,  Messenger  was  the  most  important  media  for  socialize  (with  an  average  of  41 
%),  now,  Facebook  is  the  most  important  social  media  (with  an  average  of  more  than  90  %). 

On  the  other  hand,  the  study,  also  let  us  knows  that  less  than  25%  of  the  students  had  used  virtual 
classroom.  Unfortunately,  this  average  is  too  low  if  we  consider  that  the  Campus  Azcapotzalco  released  a 
Virtual  Education  Office  in  2006  to  promote  b-learning.  This  is  disappointing  that  over  the  60  % of  our 
students  have  never  used  educational  platforms  and  only  less  than  20%  of  them  had  used  Moodle,  which  is 
our  institutional  platform  in  this  Campus. 

Considering  the  facts  mentioned  previously,  we  can  conclude  that  we  have  an  especial  situation:  students 
do  not  know  how  to  use  virtual  classroom  but,  they  are  expert  in  using  Facebook.  That  is  why  we  decided  to 
use  Facebook  as  a virtual  classroom.  All  that  inside  a b-learning  model  this  is,  to  use  virtual  classroom  and 
face-to  face-  education  model. 
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2.  ENABLING  FACEBOOK  AS  AN  EDUCATIONAL  SPACE 

Regarding  the  use  of  technology  in  academic  activities  and  according  to  the  study,  there  are  two  facts  of 
particular  relevance: 

a)  The  extraordinary  development  of  social  networks  that  has  become  a phenomenon,  especially,  for 
young  people. 

b)  The  big  potential  of  the  Virtual  Classrooms  has  not  been  used  yet  within  the  community  of  the 
UAM-A.  In  part  perhaps  because  of  the  lack  of  a good  design  of  the  interface  and  the  low  level  of  usability 
and  simplicity  that  is  required.  That  is  way  so  few  faculty  members  have  adopted  Moodle  in  their  courses. 

Based  on  the  above,  we  considered  that  it  is  necessary  to  incorporate  strategically  Social  Networks  to 
education  to  strengthen  the  contact  and  interaction  with  students  and  improve  the  education.  In  other  words, 
we  decided  to  take  the  virtual  classroom  to  the  students  instead  to  take  students  to  virtual  classroom.  With 
that  in  mind,  we  decided  to  get  more  information  on  previous  experiences  and  background  about  it.  We 
found,  at  that  time  (March  2012)  that  some  teachers  and  professors  in  the  world  had  already  used  Facebook 
to  support  virtual  communication  and  education  (for  example  Adan  Griego,  Shirly  Arrieta,  Luis  Armando 
Miramontes). 

Despite  shared  experiences  on  the  web  were  scarce,  they  allowed  us  to  drive  our  work.  We  found  that 
Facebook  allowed  us  to  create  groups  (or  "circles"  in  Google  Plus)  in  order  to  organize  our  students  to 
perform  academic  activities. 

Since  we  wanted  to  take  advantage  of  Facebook,  we  must  to  know  better  this  Social  Network  and  all  of 
the  tools  that  we  can  use,  but  first  we  had  to  answer  the  following  question:  Should  professors  and  students 
be  “friends”  in  the  Social  Network? 

We  analyzed  the  advantages  and  disadvantages  of  being  friends.  At  the  end  we  considered  that  if  we  were 
friends  of  our  students,  we  may  have  some  problems  such  as  loosing  authority  inside  the  classroom. 


3.  CREATING  THE  GROUP 

In  order  to  create  the  group  and  given  we  did  not  want  to  be  friends  of  the  students;  we  needed  to  get  two 
Facebook  profiles. 

We  created  an  “open”  group  with  my  two  different  profiles. 
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Figure  1 . My  two  different  profiles 

It  is  necessary  to  create  a URL  (Group  Address)  for  the  group  and  provide  it  to  students.  This  step  is 
essential. 
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4.  USING  FACEBOOK  TOOLS 

Facebook  offers  many  potential  tools  for  learning  like  such  as  the  possibility  to  upload  and  share,  photos, 
images,  videos,  files,  and  recently,  documents  like  word,  excel,  power  point,  PDF,  etc.  And  we  can  also 
create  “Events”,  “Ask  Question”  and  others.  Nowadays  there  are  many  educational  applications.  We  used 
“docs”  for  Facebook.  Below  we  describe  the  use  we  gave  to  each  of  these  resources. 

4.1  Files  and  Docs 

One  of  the  most  important  document  to  start  a course  is  delivering  the  study  program  to  students.  We  used 
two  ways:  “add  file”  and  the  application  “Docs”. 

The  difference  between  them  is  that  if  we  use  “Docs”,  then  our  documents  could  be  available  for  future 
uses.  Furthermore,  students  can  also  download  and  edit  them. 


Figure  2.  My  documents  in  “Docs”  for  Facebook 


We  use  this  media  to  share  the  study  program,  exercises  and  assignments,  tutorials  in  power  point,  etc. 


4.2  Add  Photos 


It  was  a very  important  resource  to  generate  sense  of  belonging  among  students.  We  took  photos  to  groups 
twice  or  three  times  during  the  course.  Actually,  they  enjoy  watching  themselves  on  Facebook. 


Figure  3.  Some  photos  of  the  students  in  the  face-to  face-class 


On  the  other  hand,  students  have  used  their  cell  phones  to  take  pictures  of  the  blackboard,  then,  we 
decided  to  do  it  as  well  and  share  them  through  Facebook.  This  was  important  to  remember  every  topic  seen 
in  class. 
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4.3  Videos 

During  the  course  were  recorded  some  videos  explaining  or  repeating  any  of  the  exercises  in  class.  This 
resource  was  useful  for  students  because  they  could  remember  all  topics  of  the  course. 


Figure  5.  An  example  of  a video  taken  during  a class 
It  is  very  important  that  the  students  produce  and  share  their  own  videos. 

4.4  Other  Tools 

Creating  events  is  useful  for  programming  exams  and  any  type  of  academic  activities  because  in  that  way 
they  will  always  be  informed. 

During  course  students  and  I had  to  generate  agreements.  We  use  Asking  Questions  and  it  was  useful  for 
it.  In  other  occasions,  we  needed  to  use  Inbox  messages  to  send  special  task  or  feedback  to  some  of  the 
students. 


5.  RESULTS 

In  general,  almost  90%  of  the  students  enrolled  had  an  active  participation.  The  platform  served  as  an 
effective  tool  for  bidirectional  communication  and  distribution  of  educational  materials  fundamentally  video 
tutorials  and  worked  examples.  Students  were  interested  in  using  this  media.  Also  it  encouraged  a sense  of 
belonging  in  the  group  and  class  participation. 


6.  CONCLUSION 

Many  changes  have  taken  place  in  society.  Social  Networks  certainly  have  changed  communication  habits, 
especially  among  young  people 

Taking  into  account  the  increase  of  use  of  Social  Networks  and  the  development  of  mobile  devices,  it  is 
necessary  to  take  advantages  about  it  and  create  new  way  to  understand  education,  as  a communicative, 
interactive  and  multimedia  process  that  is  able  to  improve  the  learning  experience. 
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Therefore,  teachers  must  generate  instructional  materials  and  digital  didactic  documents  that  encourage 
the  ingenious  and  innovative  use  of  the  ICT  in  education,  including  the  mobile  devices  as  cell  phones,  i-pods, 
tablets,  PDA,  etc.  Nevertheless,  we  must  use  them  not  only  "as  a tool  that  facilitates  the  storage,  transmission 
and  organization  of  the  courses’  content,  without  a significant  difference  between  face  to  faee  eourses  and 
others  courses  supported  by  ICT  " (Montes  et  ah,  2006),  but  also  as  a way  to  improve  the  quality  of  learning 
and  to  support  the  development  of  thought  skills  (Jonassen  and  Yueh,  1998). 

There  are  many  reasons  why  the  use  of  social  networks,  is  appropriate  for  education,  especially  higher 
education: 

a)  The  environment  of  social  media  is  widely  known  by  the  students,  so  the  learning  curve  would  be 
negligible  compared  with  other  virtual  classrooms  as  Moodle. 

b)  Facebook  provides  the  ability  for  users  to  create  groups  sectioned. 

c)  Facebook  has  a natural  tradition  for  the  use  of  photos  and  videos,  resources  that  have  great  educational 
potential. 

e)  The  ability  to  organize  and  schedule  "events". 

f)  The  ease  of  incorporating  other  educational  applications 

g)  The  growing  availability  of  mobile  phones  with  direct  access  to  social  networks,  especially  Facebook, 
Twitter  and  YouTube  contributes  to  the  u-learning.  That  is  why  we  still  believe  it  is  better  to  bring  the 
classroom  to  students  instead  of  bringing  students  to  the  classroom.  In  addition.  Social  Networks  have 
always  been  part  of  education,  even  before  the  technological  development  of  our  days. 
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ABSTRACT 

Computer-based  technologies  have  been  used  in  the  field  of  education  for  over  thirty  years.  However,  more  recently, 
powerful  and  more  affordable  mobile  technologies  are  becoming  popular  in  everyday  life  and  the  education  system.  This 
paper  reports  on  an  online  survey  of  student  body  in  a university  in  Ontario,  Canada  focused  on  the  use  of  a wide  variety 
of  digital  technologies,  including  mobile  devices  such  as  conventional  cellphones,  smartphones,  tablets,  and  e-readers. 
Preliminary  results  indicate  that  students  are  using  conventional  cellphones  with  less  frequency,  favouring  use  of 
smartphones  and  tablets  — using  such  devices  more  for  technical,  social,  and  informational  interactions  including  texting 
and  sharing  data  and  less  for  computational  functions  and  talking. 
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1.  INTRODUCTION 

Although  the  past  thirty  years  have  been  witness  to  a constant  influx  of  technology  in  the  post-secondary 
education  sector  (PSE),  attention  to  understanding  how  students  use  technology  is  far  more  recent. 
Technology  integration  in  education  has  usually  meant  teacher  use  of  technology;  however,  in  the  past  few 
years  this  technology  has  become  far  more  portable  and  affordable.  Powerful  digital  technology  is  now  in  the 
hands  of  learners  and  they  are  purchasing  and  using  it  for  a broader  set  of  purposes  (Dahlstrom,  2012). 

Mobile  devices  are  essentially  small,  handheld  computers  that  can  be  found  in  a variety  of  configurations 
including  smartphones,  tablets  and,  most  recently,  phablets,  or  intermediaries  between  smartphones  and 
tablets  (Passary,  2013).  Most  smartphones  come  equipped  with  a 8-9cm  (3-3.5”)  (measured  on  the  diagonal) 
screen  size,  while  tablets  can  have  screens  as  large  as  25-27cm  (9.7-10.6”).  Screen  resolutions  may  vary  as 
well  from  320x240  to  the  extreme  high  end  of  2048  xl536  in  the  3'^'^  Generation  iPad  with  Retina  Display 
(List  of  displays  by  pixel  density,  2013).  When  equipped  with  powerful  but  efficient  miniature  processors 
that  require  little  electrical  power  and  sufficient  memory  (16,  32,  64  and  128GB),  these  devices  can  be  relied 
upon  to  clearly  display  great  masses  of  information. 

Most  current  mobile  devices  rely  on  touch  input  interactivity  directly  on  the  screen  and  through  the  use  of 
a virtual  keyboard  that  is  displayed  on  the  screen.  Mobile  devices  may  be  equipped  with  WI-FI  or  a wireless 
local  area  network,  Bluetooth,  global  positioning  systems  (GPS)  and  cellular  radio  capabilities  that  can  allow 
connections  to  the  Internet  and  other  Bluetooth  capable  devices  such  as  an  automobile  or  a microphone 
headset.  The  feature  set  is  generally  rounded  out  with  a camera  or  two  and  a media  player  that  is  enabled  for 
video,  music  and  static  pictures.  The  operating  systems  (OS)  used  in  mobile  devices  provide  support  and 
services  for  the  use  of  small  programs  known  as  “apps”.  Originally  apps  were  created  for  a wide  variety  of 
purposes,  such  as  email,  calendaring,  and  contact  management,  and  over  the  past  2 years,  more  than  100,000 
apps  have  been  created  for  each  of  the  3 major  OS  platforms;  iOS  from  Apple,  Android  from  Google  and 
BBIO  from  Blackberry.  The  availability  of  apps  for  virtual  video  conferencing,  blogging,  microblogging  and 
other  social  networking  affordances  ensures  that  mobile  devices  will  be  used  for  educational  purposes. 
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Recently,  educational  and  commercial  institutions  and  systems  alike  have  been  affected  by  a novel 
phenomenon  referred  to  as  Bring  Your  Own  Device  (BYOD)(Hesseldahl,  2013).  Mobile  devices  are  starting 
to  be  used  in  classrooms  at  all  levels,  including  Higher  Education  (HE)  and  at  a higher  rate  by  students  than 
by  faculty,  resulting  in  an  impetus  for  a transformation  in  higher  education  in  the  areas  of  ‘recruiting, 
teaching,  learning,  delivery,  communication,  social  interactions,  and  campus  operations’  (Melton  & Kendall, 
2012). 

Today,  most  PSE  institutions  have  adopted  digital  technology  into  their  curricular  practices,  but  to 
varying  degrees,  from  simply  using  the  World  Wide  Web  as  a means  of  distributing  materials  to  students,  to 
institutions  with  full  laptop  programs.  At  the  University  of  Ontario  Institute  of  Technology  (UOIT)  in 
Oshawa,  Ontario,  Canada,  where,  for  the  past  10  years,  laptops  have  been  supplied  to  every  incoming  student 
and  where  the  environment  has  been  focused  on  facilitating  access  and  delivery  of  learning  materials  through 
the  Internet,  mobile  technology  used  by  students  is  changing  the  educational  landscape.  Students  use  their 
institutional  laptops,  but  they  are  also  seen  using  many  other  digital  devices,  especially  mobile  devices. 
These  changes  are  challenging  our  assumptions  about  these  learners  and  causing  us  to  investigate  their  use  of 
mobile  technologies  for  learning.  To  help  understanding  the  current  use  of  digital  technology  by  students  at 
UOIT,  the  study  reported  here  examined  five  research  questions:  1.  What  technologies  do  students  use?  2. 
Eor  what  purposes?  3.  How  often  do  they  use  them?  4.  How  confident  are  they  in  using  these  technologies? 
5.  How  long  have  they  been  using  these  technologies? 

In  order  to  provide  the  basic,  necessary  data,  an  instrument  was  adapted  from  a survey  designed  to  collect 
similar  data  from  teachers  and  instructors  (Desjardins  & Bullock,  2012).  Both  instruments  are  based  on  the 
premise  that  all  digital  technologies  are  fundamentally  computers,  and,  as  such,  can  only  process,  store,  and 
transmit  computer  data.  Erom  these  three  functions,  user  interactions  with  digital  technology  may  be 
categorized  into  four  general  classes  of  interactions  (Desjardins,  2005),  namely.  Technical  interactions 
(interacting  with  digital  devices).  Social  interactions  (interacting  with  others  through  digital  devices). 
Informational  interactions  (interacting  with  information  through  digital  devices),  and  Computational 
interactions  (interacting  with  data  processing  software  through  digital  devices). 


2.  METHODS 

Knowing  the  technologies  that  students  use  and  what  they  use  them  for  does  not  necessarily  indicate  whether 
they  are  able  to  use  them  effectively.  To  determine  university  students’  ability  to  use  mobile  devices,  we 
surveyed  UOIT  students  on  their  use  of  mobile  technologies  using  two  Likert  scale  indicators,  namely 
Erequency  of  Use  and  Confidence  of  Use  (Table  1). 


Table  1.  Five-point  Likert  scale  used  to  measure  frequency  of  use  and  competency  of  use 


Erequency  of  use 

Confidence  of  use 

1 = never 

2 = few  times  a year 

3 = few  times  a month 

4 = few  times  a week 

5 = daily 

1 = do  not  know  how  to  use 

2 = not  confident,  require  assistance  to  use 

3 = confident,  can  solve  some  problems 

4 = quite  confident,  can  use  with  no  assistance 

5 = very  confident,  can  teach  others  how  to  use 

Erequency  of  use  was  employed  as  an  indicator  of  effective  use  on  the  assumption  that  the  more  a specific 
device  or  system  is  used,  the  more  the  likelihood  of  use  in  different  situations  and  the  higher  the  probability 
of  reinforcing  of  procedural  knowledge.  Confidence  of  use  was  employed  as  an  indicator  of  effective  use 
since  this  measure  can  be  related  to  the  likelihood  of  a user  exploring  new  uses  and  applying  this  technology 
to  new  problems  or  tasks. 

A total  of  423  students  (n  = 423)  were  invited  by  email  to  complete  the  online  survey  and  157  students  (n 
= 157)  voluntarily  completed  the  survey,  for  a 37%  completion  rate.  Quantitative  data  gathered  from  online 
surveys  completed  by  UOIT  students  were  analyzed  in  Microsoft  Excel  using  descriptive  statistics  for 
frequency  of  use  and  confidence  in  using  mobile  technologies. 
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3.  FINDINGS 

Preliminary  data  analyzed  in  this  study  (Figure  1)  indicate  that  the  first  device  that  most  students  purchased 
were  cell  phones  without  data  plans.  However,  the  number  of  students  purchasing  their  first  cellphones 
without  data  plans  has  decreased  since  2010.  This  decrease  coincides  with  the  increase  in  the  number  of 
students  purchasing  Internet-enabled  smartphones  and  tablets  over  the  same  period. 


Figure  1.  Year  of  first  and  last  purchase  of  cellphones  (no  data),  smartphones  (with  data),  and  tablets 

Of  the  four  general  categories  of  devices  (cellphones  with  data  plans,  cellphones  without  data  plans, 
smartphones  with  data  plans,  and  tablets),  students  had  the  most  access  to  Internet-enabled  smartphones,  with 
the  majority  of  these  students  owning  at  least  one  smartphone.  The  number  of  students  with  access  to  a tablet 
or  e-reader  was  higher  than  the  number  with  access  to  cell  phones  with  no  data  plan. 

150  -| 

■ I own  at  least  one  □ Don't  own  but  have  access  to 


Cell  Phone  (no  data  plan)  Smartphone  Tablet/Pad/e-Reader 

Figure  2.  Student  access  to  cellphones  (no  data),  smartphones  (with  data),  and  tablets 

Data  in  Table  2 indicate  that  students  used  their  smartphones  most  frequently  (daily)  to  communicate  via 
texting,  with  other  modes  of  communication,  namely  audio,  email,  and  social  media  occurring  reasonably 
frequently  (a  few  times  a week).  These  students  were  quite  confident  in  their  abilities  to  use  their 
smartphones  for  these  purposes.  Students  also  used  their  smartphones  to  create  and  edit  documents  and 
multimedia  objects  reasonably  frequently.  Students  used  their  smartphones  less  frequently  to  search  for 
information,  although  they  were  confident  in  being  able  to  obtain  this  information  with  these  devices.  Most 
students  did  not  use  their  smartphones  for  computational  purposes;  this  was  reflected  in  their  lower 
confidence  levels  for  these  tasks.  With  the  exception  of  sharing  documents,  students  primarily  used  their 
smartphones  for  technical,  social,  and  informational  interactions  and  mainly  for  personal  purposes. 
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Table  2.  Frequency,  confidence,  and  main  purpose  of  smartphone  use  for  the  four  classes  of  interactions. 


Frequency 
of  use 

Confidence 
of  use 

Main  purpose  of  use 

(5-point  scale) 

(5-point  scale) 

Personal 

Studies 

Work 

Create/edit  documents 

3.66 

4.33 

78.4% 

14.8% 

6.8% 

u 

'c 

u 

01 

Create/edit  voice  recordings 

2.19 

4.02 

69.2% 

23.1% 

7.7% 

1- 

Create/edit  multimedia 

3.14 

4.33 

85.5% 

8.1% 

6.5% 

Email 

3.92 

4.69 

65.1% 

12.7% 

22.2% 

Video 

2.21 

4.33 

92.5% 

0.0% 

7.5% 

Texting 

4.76 

4.74 

95.7% 

1.1% 

3.2% 

Social 

Audio 

3.86 

4.62 

93.2% 

1.4% 

5.4% 

Social  media 

3.80 

4.62 

98.5% 

0.0% 

1.5% 

Share  document 

1.97 

4.02 

18.2% 

69.7% 

12.1% 

Share  works  and  ideas 

1.55 

3.68 

66.7% 

20.0% 

13.3% 

Search  for  articles 

2.68 

4.57 

63.9% 

27.8% 

8.3% 

<T3 

C 

Search  for  short  videos 

2.99 

4.73 

86.5% 

9.6% 

3.8% 

o 

<T3 

E 

Search  for  movies 

1.25 

3.60 

100.0% 

0.0% 

0.0% 

o 

c 

Search  for  music 

1.59 

3.84 

100.0% 

0.0% 

0.0% 

Search  for  ebooks 

1.34 

3.53 

85.7% 

14.3% 

0.0% 

Share  calendar 

2.95 

4.18 

59.5% 

29.7% 

10.8% 

Use  concept  maps 

1.05 

2.94 

33.3% 

66.7% 

0.0% 

c 

o 

Use  diagrams 

1.17 

2.74 

42.9% 

28.6% 

28.6% 

<T3 

Z5 

Q. 

£ 

o 

u 

Sort  data 

1.25 

3.26 

80.0% 

10.0% 

10.0% 

Produce  graphs 

1.14 

3.00 

14.3% 

57.1% 

28.6% 

Complex  calculations 

1.71 

3.88 

55.0% 

40.0% 

5.0% 

4.  CONCLUSIONS 

Preliminary  analyses  of  the  data  in  this  study  indicate  that  the  majority  of  post-secondary  students  who 
participated  in  this  study  changed  the  types  of  mobile  devices  they  have  been  using  since  2010,  reducing 
ownership  of  cellphones  without  data  plans  in  favour  of  smartphones  and  tablets  — primarily  employing 
these  devices  for  technical,  social,  and  informational  interactions.  Furthermore,  initial  findings  indicate  that  a 
majority  of  these  students  own  at  least  one  smartphone  and  more  have  access  to  a tablet  or  e-reader  than  a 
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cell  phone  with  no  Internet  accessibility.  These  analyses  also  indicate  that  students  are  using  their  mobile 
devices  more  for  texting  and  sharing  data  than  for  talking  only.  These  results  point  to  a marked 
transformation  in  student  frequency  of  use  and  confidence  in  using  Internet-accessible  mobile  devices  such  as 
smartphones  and  tablets  (Melton  & Kendall,  2012),  and  suggests  further  studies  focused  on  student  and 
professor  use  of  mobile  devices  in  the  learning  and  teaching  process. 
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ABSTRACT 

The  purpose  of  this  study  is  to  probe  into  the  effect  of  metacognitive  thinking  strategies  with  different  teaching  strategies 
in  an  online  inquiry-hased  junior  high  course.  Two  types  of  teaching  strategies  were  applied  in  classes  with  totally  69 
students  of  9*  graders  in  New  Taipei  City.  Among  participants,  34  students  were  in  the  experimental  group  accepting  the 
strategy  of  “Collaborative  Online  Inquiry”  while  the  control  group  of  35  students  with  “Personal -based  Online  Inquiry”. 
Data  were  collected  for  a six-week  lesson  on  this  topic  “Campus  Folk  Song”.  ANCOVA  and  class  observations, 
instructors’  diary,  and  interviews  with  students  are  measured  to  understand  the  different  effects  of  two  teaching  strategies 
on  students’  metacognitive  thinking  strategies.  The  result  shows  that  students  by  ‘collaborative  online  inquiry’  in 
experimental  group  perform  their  metacognitive  strategies  better  than  those  by  “Personal-based  Online  Inquiry”  in  the 
control  group. 
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1.  INTRODUCTION 

“Campus  Folk  Song”  is  popular  during  the  1970s  to  1990s  in  Taiwan.  The  music  style  is  similar  to  country 
song  but  with  more  local  consciousness.  They  were  originated  from  the  student  singers  in  colleges  and  those 
song  revealed  the  inner  voices  of  the  man  in  the  street  and  some  aroused  many  deep  thoughts  about  the 
Taiwan  positioning  in  world  politics  and  it  was  resonatory  in  that  times. 

Though  “Campus  Folk  Song”  represents  a period  of  historical  moment  of  Taiwan  democracy  progress 
and  was  with  great  impact  on  ordinary  people  in  many  respects,  it  is  hard  to  catch  the  attention  of  junior  high 
students,  not  to  mention  to  investigate  in  further  the  historical  background,  famous  composers,  famous  songs, 
etc.  In  order  to  motivate  the  net  generation  to  research  this  topic,  online  inquiry  learning  strategy  were  used 
in  the  class.  Wallace  and  Kupperman(2000)  indicate  that  online  information  seeking  is  a complex  and 
difficult  process.  The  intension  to  develop  students'  understanding  of  content  through  use  of  the  Internet  is  a 
challenging  task  for  both  students  and  teachers  though  this  learning  strategy  may  more  attractive  to  the  digital 
natives.  Because  Internet-based  learning  environment  is  with  high  degrees  of  freedom,  it  may  favors  students 
who  possess  mature  computer  skills  and  metacognitive  skilIs(Park  et  al.  2009).  Tsai(2009)  reported  that 
students’  metacognitive  strategies  play  important  roles  in  their  online  inquiry  learning,  and  Huang,  Shen  and 
Chang(2011)  also  pointed  out  that  promote  metacognitive  thinking  is  very  important  when  using  online 
inquiry  learning. 

The  metacognitive  activities  involved  in  online  inquiry  include  metacognitive  knowledge  and 
metacognitive  regulation.  Metacognitive  knowledge  is  knowledge  about  one’s  self  as  a learner,  task 
knowledge  and  strategic  to  learn(Flavel,1979;  McCormick, 2003).  While  metacognitive  regulation  of  online 
inquiry  includes  (l)planning  the  online  inquiry;  (2)monitoring  and  controlling  the  progress  through  the  online 
inquiry  process;  and  (3)reflecting  on  what  was  learned  after  reading  certain  information(Schraw,1998; 
Howard  et  al.  2011). 

In  order  to  help  teachers  facilitate  students’  learning  and  to  ensure  learners  are  really  involved  in  higher 
levels  of  cognitive  activities  rather  than  just  copy  and  paste  aggregately,  this  study  used  collaboration  as  a 
teaching  strategy  to  stimulate  learners  metacognitive  thinking,  and  the  effect  on  metacognition  performance 
of  online  inquiring  learning  is  examined. 
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2.  RESEARCH  METHOD 

2.1  Participant 

A pretest-posttest  quasi-experimental  design  was  executed  in  a New  Taipei  City’s  junior  high  school. 
Participants  were  sixty-nine  students  among  two  classes.  One  class  with  thirty-four  students  was  assigned  to 
the  experimental  group,  and  another  class  with  thirty-five  students  was  assigned  to  the  control  group.  They  are 
ninth  graders  and  their  families  are  with  ordinary  social  status. 

2.2  Instructional  Design  of  Online  Inquiry  Learning 

A six-week  online  inquiry  learning  on  “Campus  Folk  Song”  was  held  in  two  classes.  Basically  the  online 
inquiry  journey  had  4 steps:  planning  the  web  inquiry  strategy,  searching  web  resources,  analyzing  & 
synthesizing,  evaluating.  Researchers  designed  four  questions  to  guide  students’  investigating  activities  and 
the  online  inquiring  steps  were  scaffold  through  learning  sheets  which  were  distributed  in  the  beginning  of 
each  class.  Two  classes  of  students  should  finfish  their  own  learning  sheets  and  submit  in  the  end  of  each 
class.  Besides,  students  in  experimental  group  needed  to  finish  one  extra  group  activity  learning  sheet. 


Table  1.  Instructional  Design  Outline 


Cycle 

Questions  to  answer  and  Online  Inquiry  Process 

7ycle  1 

Questions  to  investigate  in  cycle  1 : 

What  is  the  definition  of  “Campus  Folk  Song”? 

Personal-based 
Online  Inquiry 
(control  group) 

Planning  the  web  inquiry 
strategy 

Searching 

web 

resources 

Analyzing  & synthesizing 

Evaluating 

Collaborative 
Online  Inquiry 
(experimental 
group) 

Collaborative 

Discussing 

Planning  the 
web  inquiry 
strategy 

Searching 

web 

resources 

Analyzing  & 

synthesizing 

Collaborative 

Discussing 

Evaluating 

Collaborative 

Discussing 

7ycle  2 
7ycle  4 

Questions  to  investigate  in  cycle  2: 

Why  “Campus  Folk  Song”  is  so  popular  during  1970~1990  in  Taiwan? 
Questions  to  investigate  in  cycle  3: 

What  are  the  style  and  lyric  characteristics  of  “Campus  Folk  Song”? 
Questions  to  investigate  in  cycle  4: 

Are  there  some  typical  composer  or  singer  of  “Campus  Folk  Song”  ? 

Personal-based 
Online  Inquiry 
(control  group) 

Planning  the  web  inquiry 
strategy 

Searching 

web 

resources 

Analyzing  & synthesizing 

Evaluating 

Collaborative 
Online  Inquiry 
(experimental 
group) 

Collaborative 

Discussing 

Planning  the 
web  inquiry 
strategy 

Searching 

web 

resources 

Analyzing  & 
synthesizing 

Collaborative 

Discussing 

Evaluating 

Collaborative 

Discussing 

!^ycle5 

Task  to  complete: 

Prepare  an  oral  presentation  in  front  of  the  class  with  ppt. 

Personal-based 
Online  Inquiry 
(control  group) 

Evaluating 

Planning  the  web 
inquiry  strategy 

Searching  web  resources 

Analyzing  & synthesizing 

Collaborative 
Online  Inquiry 
(experimental 
group) 

Evaluatin 

g 

Collaborative 

Discussing 

Planning  the  web 
inquiry  strategy 

Searching 

web 

resources 

Analyzing  & 
synthesizing 

Collaborative 

Discussing 

Work  on 
group 

presentation 

ppt 

I'ycle6 

Task  to  complete: 

Deliver  the  presentation  and  finish  the  peer  evaluation  sheet. 

Personal-based  Online  Inquiry  (control  group) 

Oral  presentation,  evaluating  , discussing  and  reflecting 

Collaborative  Online  Inquiry  (experimental  group) 

Oral  presentation,  evaluating  , discussing  and  reflecting 

390 


lADIS  International  Conference  e-Learning  2013 


2.3  Instruments 

2.3.1  Metacognitive  Strategy  Scale  for  Online  Inquiry  Learning 

The  Metacognitive  Strategy  Scale  for  Online  Inquiry  Learning  (MSSOIL)  used  in  this  study  was  developed  by 
Huang(201 1)  which  was  developed  based  on  Metacognition  Rating  Scale  for  General  Biology  (MRSGB)  and 
Metacognive  Self-regulation  sub-scale  of  Motivated  Strategies  for  Learning  Questionnaire(MSLQ).  The 
MSSOIL  is  a 6-point  Likert  scale  reflecting  students’  metacognitive  behavior  when  working  on  their  online 
inquiry  learning.  The  overall  Cronbach’s  coefficient  alpha  of  MSSOIL  was  .91  while  the  subscales’  are  from 
.75  to  .87.  The  dimension  description  of  MSSOIL  is  presented  in  the  following  table. 


Table  2.  Description  of  MSSOIL’S  Dimensions 


Dimension 

Item 

alpha 

Self-plan 

(SP) 

(4  items) 

• In  terms  of  doing  the  online  inquiring  assignment,  students  have  some 
thoughts  about  the  purposes,  searching  strategies,  reading  skills  before 
browsing  the  web 

.82 

.91 

Learning-monitor 

(LM) 

(4  items) 

• Students  usually  think  about  the  relationship  between  the  course 
content  and  the  information  on  the  web,  and  also  the  connections 
between  the  assignment  project  and  the  information  on  the  web. 

.87. 

Comprehension 
monitor(CM) 
(3  items) 

• Students  usually  know  whether  there  is  anything  not  understood  yet 
when  browsing  the  web,  and  try  to  make  sense  to  information  which  is 
usable  but  unfamiliar. 

.77 

Self-modify 

(SM) 

(7  items) 

• Students  can  usually  find  a better  way  to  search  for  appropriate 
information  and  adjust  personal  views  toward  project  topic  if  there  is 
problem  to  complete  the  job. 

.85 

Self-evaluate 

(SE) 

(4  items) 

• Students  usually  try  to  examine  the  proficiency  of  knowledge  acquired 
after  navigating  the  web,  and  they  also  have  interests  to  review  peers’ 
comments  if  possible. 

.75 

3.  FINDINGS 


The  results  of  covariance  in  metacognition  performance  are  showed  Table  3 and  table  4 which  summarize  the 
regression  homogeneity  test  and  ANCOVA  respectively  for  “Collaborative  Online  Inquiry”  group  and 
“Personal-based  Online  Inquiry”  group  in  the  metacognition  performance  of  online  inquiry  learning. 


Table  3.  Summary  of  Regression  Homogeneity  Test  For  Both  Teaching  Strategies  In  Metacognition  Performance 


Dimension 

Sources  of  variance 

SS 

df 

MS 

F 

Significance 

Total 

Between(regression-coefficient) 

66.914 

1 

66.914 

.538 

.466 

Within  (error) 

8079.467 

65 

124.299 

SP 

Between(regression-coefficient) 

4.264 

1 

4.264 

.651 

.423 

Within  (error) 

425.664 

65 

6.549 

LM 

Between(regression-coefficient) 

4.798 

1 

4.798 

.630 

.430 

Within  (error) 

495.113 

65 

7.617 

CM 

Between(regression-coefficient) 

1.186 

1 

1.186 

.224 

.637 

Within  (error) 

343.762 

65 

5.289 

SM 

Between(regression-coefficient) 

41.511 

1 

41.511 

2.354 

.130 

Within  (error) 

1146.121 

65 

17.633 

SE 

Between(regression-coefficient) 

1.788 

1 

1.788 

.230 

.633 

Within  (error) 

504.689 

65 

7.764 

P<.05 


According  to  Table  3,  the  regression-coefficient  homogeneity  test  did  not  show  statistical  significance  in 
the  total  scores  (F=0.538,  p>.05)  and  each  dimension.  This  means  that  these  two  teaching  strategies  have  the 
same  slope  and  conform  to  the  fundamental  assumption  within  regression-coefficient  homogeneity. 
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Table  4.  Summary  of  ANCOVA  For  Both  Teaching  Strategies  In  Metacognition  Performance 


Dimension 

Sources  of  variance 

SS 

df 

MS 

E 

Significance 

Total 

Between(group) 

1469.813 

1 

1469.813 

11.908 

.001“ 

Within  (error) 

8146.381 

66 

123.430 

SP 

Between(group) 

11.319 

1 

11.319 

1.738 

.192 

Within  (error) 

429.928 

66 

6.514 

LM 

Between(group) 

85.483 

1 

85.483 

11.286 

.001“* 

Within  (error) 

499.911 

66 

7.574 

CM 

Between(group) 

29.655 

1 

29.655 

5.674 

.020** 

Within  (error) 

344.948 

66 

5.226 

SM 

Between(group) 

200.401 

1 

200.401 

11.137 

.001“* 

Within  (error) 

1187.632 

66 

17.994 

SE 

Between(group) 

34.702 

1 

34.702 

4.522 

.037** 

Within  (error) 

506.477 

66 

7.674 

*P<.05,  ”P<.01 


The  ANCOVA  results  in  Table  4 exclude  the  influence  of  original  ability  (pre-test),  and  the  two  teaching 
strategies  in  the  metacognition  performance  assessment  indicated  that  experimental  group  made  significantly 
greater  gains  compared  with  control  group  in  comprehension-monitor  (F  (1,  66)  = 11.286,  p <0.01),  self- 
modify  (F  (1,  66)  = 5.674,  p < 0.05),  self-evaluation  (F  (1,  66)  = 11.137,  p < 0.01),  cross-evaluation  (F  (1, 
66)  = 4.522,  p<  0.05)  and  total  scores  (F(l,  66)= 11.908,  p<0.01).  The  dimension  of  self-plan  (F  (1,  66)  = 
1.738,  p >0.05)  is  not  significantly  different  in  the  post  test  excluding  the  pre-test  effect. 


4.  DISCUSSION  AND  CONCLUSION 

In  the  “Learning-monitor”  dimension,  students  in  the  experimental  group  were  more  aware  of  their  learning 
process  because  they  had  more  chances  to  interact  with  others  to  discuss  the  relationship  between  the 
questions  teacher  proposed  and  the  information  they  inquired  on  the  web.  For  the  “Comprehension-monitor” 
dimension,  groups  with  collaborative  discussion  performed  better  for  they  need  to  clarify  the  usefulness  of 
information,  and  by  went  through  this  process  they  explained  what  they  learned  from  the  web  messages 
reciprocally  which  can  help  them  to  be  more  reflective  on  their  comprehension  level  of  this  learning  topic.  As 
the  “Self-modify”  and  “Self-evaluate”  dimensions,  students  in  the  experimental  group  faced  more 
compulsion  to  listen  to  others  and  group  activity  also  provided  more  chances  to  communicate  different 
opinions  which  can  both  encourage  and  force  learners  to  evaluate  and  modify  their  learning. 

Whipp  et  al.(2004)  suggested  that  proper  self-monitor  and  tracking  are  important  characteristics  of 
computer-based  learning,  because  students  always  confront  with  metacognitive  challenges  about  task 
understanding,  planning,  monitoring,  regulation,  and  reflection  throughout  the  whole  online  inquiry  process 
(Quintana  et  al.  2004).  In  this  study,  collaborative  online  inquiry  learning  presented  better  metacognition 
performance  than  personal  based  online  inquiry  significantly.  This  result  provides  a confirmation  of  positive 
effect  of  collaboration  on  metacognitive  thinking,  and  further  studies  about  the  interaction  during  the 
collaboration  are  suggested. 
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ABSTRACT 

This  article  deals  with  nowadays  urgent  issue.  It  tries  to  find  a way  how  to  achieve  as  highest  probability  of  current 
students  employment  as  possible,  especially  in  the  age  of  business  crises.  It  comes  from  actual  industry  practice 
requirements  on  hiring  employees.  There  is  briefly,  considering  limited  range  of  article,  described  innovative  education 
concept  “Educational  company”,  which  implies  elements  of  problem  based  learning  with  focus  on  e-learning  application 
and  acquirement  of  soft  skills  by  students.  This  article  shows  structure  of  mentioned  concept.  There  are  mentioned 
differences  between  traditional  and  developed  education  concept.  A contribution  describes  current  experiences  of  all 
participants,  concept  benefits  and  also  problems,  which  this  concept  faces.  There  are  presented  further  steps  connected 
with  concept  development. 

The  experience  of  concept  versatility  are  very  good  and  show  it's  high  potential.  A concept  is  motivating  not  only  for 
students  but  also  for  the  teachers,  which  are  acquiring  the  new  abilities  and  are  getting  new  knowledge.  Mostly  it  is  going 
about  more  effective  form  of  education  with  clear  results  not  only  in  the  form  of  created  products  but  mainly  in  good 
student  versatility  in  practice. 
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1.  INTRODUCTION 
1.1  The  Theoretical  Outlets 

Education  at  all  types  of  schools  should  go  through  regular  innovations.  The  reason  is  achieving  it's  quality 
results.  Schools  are  operating  in  a very  competitive  environment  like  all  other  organizations.  In  this,  it  is 
necessary  to  help  graduated  students  in  the  future  versatility  in  the  globalized  labor-market.  The  process  of  an 
education  includes  important  factors  such  as:  teacher,  student,  subject  or  specialization,  etc.  Each  student  has 
his  way  of  learning.  It  includes  perception  and  processing  of  knowledge  (Kohoutek,  2008).  During  an 
education  the  method  of  student  learning  affects  the  teacher's  teaching  (Prucha,  2009). 

The  requirements  for  specialization  by  students  are  rising  according  to  (Smith,  2005).  On  the  other  hand, 
companies  require  from  the  schools  greater  connection  with  practice  and  mastery  of  soft  skills  as  well.  Also 
these  requirements  decide  about  student  versatility  of  labor  market  (Meyers,  2007).  Therefore,  also  in  the 
Czech  Republic  promote  educational  systems  of  various  types.  Except  from  traditional  teaching  “in  front  of 
blackboard”  are  discussed  other  learning  systems,  above  all  e-learning  and  Process  Based  Learning. 
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E-learning  is  a form  of  education  that  uses  informational  technologies  to  inform  the  students,  to  provide  a 
communication  between  student  and  teacher  or  more  students  together  and  for  the  distribution  of  learning 
materials  and  presentation  of  the  theory  and  also  the  management  of  education  (Dostal,  2008).  The  e-learning 
materials  are  often  created  in  multimedia  forms  (see,  e.g.  Simon  and  Edl,  2011;  Edl,  2012).  Implementation 
of  e-learning  takes  place  through  the  Learning  Management  System.  Therefore,  the  smart  learning  systems 
have  a three  program  modules  - teachers  (it  includes  a specifics  of  a teaching  materials  and  theory),  student 
(it  includes  a specifics  of  a concrete  student),  and  tutor  (it  includes  teaching  strategies)  according  (Brdicka, 
2004).  Process  Based  Learning  is  the  education  concept  of  larger  and  longer-term  issues  that  requires  the 
active  exploration  by  students.  Compared  to  the  traditional  education  this  concept  is  less  structured. 
However,  students  must  work  together,  organize  their  work  and  lead  to  the  desired  result  (Brownell  et  al., 
2004). 

To  the  development  of  these  mentioned  methods  in  terms  of  empirical  research  would  like  to  contribute 
the  authors  of  this  article  as  well. 

1.2  The  Practical  Outlets 

Unrelenting  market  requirements  and  tough  competition  are  reflected  in  the  personnel  area  of  manufacturing 
organizations.  Companies  choose  their  employees  with  not  only  their  professional  knowledge  in  the  given 
field  (such  as:  the  construction  machinery  or  preparations,  the  technologies  of  plastics  processing  and  mold 
manufacturing,  etc.)  and  language  training.  Leading  organizations  are  beginning  to  realize  the  importance  of 
soft  skills. 

Industry  according  the  author  (Manlig,  2010)  with  particular  emphasis  on: 

• the  comprehensive  interdisciplinary  knowledge, 

• the  processing  thinking, 

• the  systematic  approach, 

• the  ability  to  work  in  the  team, 

• the  practical  experience  in  the  branch. 

In  addition  in  the  present,  organizations  are  required  from  the  workers,  according  to  available  job  offers: 
interdisciplinary  collaboration,  time  management,  communication  skills,  independence,  creativity,  accuracy, 
reliability,  ability  to  work  with  legislation  and  internal  business  documents.  Workers  must  be  able  to  control 
a variety  of  information  technologies  (software  for  various  fields/branches).  They  are  accessing  the 
capabilities  to  introduce  a modern  technologies  and  orientation  in  various  tools  of  management. 

Most  companies  already  aren't  looking  for  only  the  specialists  (exclusive  experts  in  the  given  area),  but 
also  interdisciplinary  professionals  who  are  able  to  solve  business  ties  and  problematic  tasks  much  more 
complexly  - they  have  the  procedural  and  holistic  thinking.  Therefore  technically  oriented  engineers  with  the 
comprehensive  knowledge  (i.e.  technically  oriented  multi-professional  worker  who  knows  the  business 
processes  and  the  ties  between  them)  find  out  the  versatility  in  the  present  time. 


2.  IDEA  OF  EDUCATIONAL  CONCEPT  „EDUCATIONAL  COMPANY“ 

The  idea  of  educational  concept  of  "Educational  Company"  based  on  the  mentioned  requirements  of 
industrial  practice.  It  introduces  the  elements  of  problem-oriented  education  and  e-learning  to  the  education, 
as  are  presented  in  detail  in  the  text  above.  To  date,  there  is  a significant  gap  between  business  requirements 
for  recruitments  and  knowledge  taught  in  schools  of  all  types  in  the  country.  The  presented  concept  is  trying 
to  close  this  gap. 

The  concept  of  "Educational  Company"  connects  the  selected  objects  into  an  unified  education  system  by 
complex  project.  In  this  project,  the  students  solve  the  problems  related  to  the  design  of  batch  production.  It 
proceeds  from  the  initial  idea  and  a introductory  marketing  study,  over  the  development  of  the  product,  to  the 
design  of  a production  in  the  concrete  shop  floor  conditions  of  the  home  department,  as  is  illustrated  in 
Eigure  1. 


395 


ISBN:  978-972-8939-88-5  © 2013  lADIS 


Development 


The  complex  view  on 
company  processes^ 


Production  planning 


Technology  and 
NC  programming 


Business  plan 


Quality  control 
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Figure  1 . Schematic  concept  of  „Educational  Company" 


In  addition  to  getting  the  expertly  knowledge  the  students  learn  to  work  in  a team.  They  acquire  the 
current  and  newly  developed  company  ties.  They  learn  to  work  considerately  of  others'  work.  Quality 
permeates  of  the  all  performed  activities.  Above  solved  problematic,  the  people  discuss  in  personal  contact 
and  through  information  technology  as  well,  and  not  only  in  limited  time  of  teaching.  On  the  other  hand, 
must  acquire  soft  skills.  This  is  connected  with  the  deliberate  application  of  information  technologies  and  the 
involvement  of  "common  sense"  and  humanity,  which  is  in  the  some  areas  currently  being  lost.  The  concept 
is  trying  to  balance  the  aspects  of  the  technology  world  and  emphasis  on  grasping  of  the  human  factor  in 
modern  business. 

The  concept  of  "Educational  Company"  with  its  structure  also  builds  on  the  idea  of  the  process,  as  is 
currently  enforced  in  quality  control.  The  output  of  one  object  is  the  input  to  another  object  (e.g.  design  of  the 
technology  for  specific  departmental  conditions  in  the  study  subject  CAD  / CAM  (Computer  Aided  Design  / 
Computer  Aided  Manufacturing)  is  the  output  at  the  same  time  it  is  the  input  for  the  design  of  production 
process  in  the  study  subject  Design  of  the  production  systems).  Here  is  the  aspiration  to  make  students 
working  on  the  project  very  similar  to  real  production  conditions.  In  this  context,  regular  consultations  with 
industrial  companies  in  the  area  are  utilized.  Even  there  is  an  aspiration  to  introduce  partial  cooperation  in  the 
"sharp"  business  problems  of  related  organizations.  This  places  big  demands  on  all  involved  partners. 

Eor  a closer  approximation  in  the  following  table  (Table  1)  are  shown  the  main  differences  between  the 
innovative  concept  of  "Educational  Company"  and  the  traditional  education  of  "In  front  of  blackboard", 
which  is  very  common  not  only  in  this  country. 
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Table  1.  The  comparing  of  "Educational  Company"  with  "In  front  of  blackboard" 


Activity 

Traditional  education 
„In  front  of  blackboard“ 

„Educational  Company“ 

Course  interdependence 

Low 

Aspiration  for  the  biggest  connect  object 

Task  solving 

Tasks  are  solved  separately  in  each  course 

Complex  project  - solving  task  from  the 
perspective  of  different  profession 

Types  of  projects 

Given  by  nature  of  the  project  - from 
individual  to  team  projects 

Team  project  with  emphasis  on  project 
management  principles 

Type  of  task 

Mostly  model  situations 

Solution  design  for  specific  workshop 
conditions 

Pedagogues’  role  during 
project  solving 

Superior  behaviour  and  task  control 

Usually  project  head  manager  (sponsor), 
can  behave  also  as  a coach 

Student  motivation 

Depends  on  the  type  of  task/project  low 
till  high 

High 

Demand  placed  on  teacher  to 
prepare  course 

Low  till  medium 

High 

Use  of  information 
technologies 

Low  till  medium 

Medium  till  high 

Use  of  teaching  aid  materials 

Small 

High,  rich  production 

Requirements  of  course 

Low 

High,  continually 

Soft  skills  learning 

Unnoticeable 

Rich 

3.  CONCLUSION 

Past  experience  with  the  introduction  of  the  concept  of  "Educational  Company",  both  from  the  teachers  and 
the  positive  response  of  graduates  and  other  professionals  from  the  industry,  demonstrate  the  considerable 
potential  of  this  approach  to  education.  An  unique  connection  of  normal  professional  knowledge  versatility, 
information  technologies  and  soft  skills,  as  well  as  the  creation  of  additional  knowledge,  are  preparing 
students  for  the  real  situation  not  only  in  domestic  organizations. 

In  addition  to  professional  knowledge  is  highlighting  in  particular  following: 

• scope  and  knowledge  of  corporate  ties, 

• easier  integration  of  graduates  into  business  processes. 

Finally,  there  is  outlined  next  development  of  described  concepts: 

• creating  of  e-learning  support  for  particulars  modules  of  "Educational  Company"  concept, 

• the  expansion  of  an  existing  example  of  the  design  production  process  as  well  as  learning  example 
for  complex  design  of  a new  production  machine  concept  with  regard  to  both  technical  and 
technological  capabilities,  as  well  as  organizational  requirements, 

• currently  we  are  discussing  about  the  possibilities  of  linking  education  between  some  subjects  and 
faculties. 
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The  aim  of  the  cooperation  of  the  Faculty  of  Mechanical  Engineering  with  the  Faculty  of  Economics  is 
bigger  interconnection  of  technical  aspects  of  the  project  with  economic  aspects. 

Significant  involvement  of  the  other  faculties  still  alludes  to  well-established  ways  of  education,  which 
would  be  "very  difficult  to  change".  Inter-faculty  cooperation  is  therefore  solved  directly  with  personal 
contact  (staff)  across  the  university. 

Application  of  the  "Educational  Company"  concept  requires  a good  teachers  and  also  experts  in  given 
fields.  Not  only  knowledge  is  needed  but  also  technical  support  and  finances.  In  this  way,  the  education  is 
related  especially  with  necessary  willingness  and  the  enthusiasm  of  the  teachers.  This  is  the  motor  of  this 
difficult  but  interesting  project. 
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ABSTRACT 

This  position  paper  deploys  a dialectical  worldview  and  a critical  sociological  perspective  on  practice  to  conceptualize 
professional  e-leaming  as  a reflexive  bundle  of  knowing,  doing,  thinking  and  reflecting  that  produces  practical 
knowledge  as  the  power  to  change  when  immersed  in  the  dynamism  of  practice  across  social  structures.  It  is  therefore  a 
structural  constructivist  process.  Two  implications  for  the  design  of  electronic  continuing  professional  development 
(eCPD)  pedagogical  models  to  address  this  learning  challenge  so  as  to  improve  the  impact  of  eCPD  programmes  on 
professional  practice  are  explored.  Firstly,  acknowledging  the  diversity  of  social  actors  in  any  process  of  eCPD  for 
changing  practice  highlights  that  implementing  technologies  and  implementing  new  evidence  to  improve  practice  brings 
up  issues  of  conflict  and  confrontation  because  of  the  agency  of  all  practitioners.  Secondly,  in  such  a context,  educators 
could  tackle  the  challenge  of  managing  the  reflexivity  of  professional  networks  by  creating  new  informal  social  structures 
for  social  interaction  and  collaborative  learning  networked  with  technologies  that  disrupt  existing  structures.  Finally,  it 
concludes  with  recommendations  for  designing  eCPD  as  a strategy  to  improve  impact  on  practice. 


KEYWORDS 

Agency,  CPD,  epistemology,  professional  e-leaming,  online  pedagogical  models,  reflexivity 


1.  INTRODUCTION 

Today,  electronic  continuing  professional  development  (CPD)  is  routinely  delivered  to  various  professionals 
across  the  world  through  Internet-connected  devices  such  as  computers,  laptops,  smartphones  and  tablets. 
Whether  for  doctors,  nurses,  teachers,  engineers,  accountants  or  filmmakers,  eCPD  courses  largely  take  the 
form  of  self-study  modules  with  videos  and  podcasts,  as  well  as  interactive  blogs  and  discussion  forums. 
eCPD  aims  to  support  professionals  update  knowledge  and  skills,  as  well  as  share  and  collaborate,  so  as  to 
improve  practice,  including  the  products  and  services  offered  to  end  users  such  as  patients  in  healthcare.  Yet, 
despite  critical  reviews  showing  the  low  impact  of  current  eCPD  pedagogical  models  from  current  research 
in  healthcare  (author  et  ah,  forthcoming),  there  are  few  attempts  to  rethink  the  underlying  epistemological 
assumptions  of  designing  eCPD  pedagogical  models.  Current  research  into  designing  and  evaluating 
innovative  eCPD  approaches  in  healthcare  to  improve  impact  on  practice  (author  et  al,  2012)  tackles  this 
problem  by  reframing  the  epistemology  for  professional  e-learning.  This  position  paper  elucidates  a new 
epistemology  of  professional  e-learning  - structural  constructivism,  critically  discusses  two  key  implications, 
and  makes  recommendations  for  designing  eCPD  to  manage  and  optimize  the  ‘e-learning-practice  change’ 
relationship  and  improve  impact. 


2.  REFLEXIVITY:  PROBLEMATISING  EXISTING  eCPD  DESIGN 

From  a critical  perspective  on  education,  existing  pedagogical  models  for  eCPD  based  on  a set  of  'objectivist' 
and  'socially  constructed'  assumptions  about  professional  e-learning  that  are  neither  'objectivist'  nor  'socially 
constructed'  are  a problem.  They  cast  professionals  who  learn  reflexively  - that  is  they  learn  automatically 
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from  experience  - to  improve  their  practice  through  informal  social  relationships  (Gabbay  & LeMay,  2004; 
Nardi  et  al,  2002)  as  a problem  to  be  fixed  with  inputs  of  knowledge  and  skills,  and  assume  a static  linear 
causal  relationship  between  e-learning  inputs  and  practice  improvement  outputs.  While  social  constructivism 
promises  much  in  theory,  its  enactment  in  online  environments  to  deliver  short  courses  to  busy  professionals 
is  challenging  (Nunes  & McPherson,  2007;  Sandars  et  ak,  2007).  Social  constructivist  pedagogical  models  do 
not  change  or  modify  the  wider  structures  that  produce  unanticipated  barriers  to  professionals’  capacity  for 
agency  to  change  practice  as  a result  of  learning  on  an  eCPD  programme.  While  acquisition  and  participation 
are  both  important  in  learning  (Sfard,  1999),  as  are  critical  reflection  and  practitioner  research  (Brookfield 
1987;  Carr  & Kemmis,  1986;  Schon,  1983)  the  impact  on  improving  practice  ultimately  rests  on  the  ability  of 
pedagogical  models  to  improve  the  access  of  professionals  to  a multiplicity  of  perspectives  and  resources  in 
in  safe  spaces  such  as  informal  networks  to  work  on  and  try  out  new  practices  that  empower  them.  The 
danger  of  current  binary  debates  about  ‘what  works’  in  eurriculum  development  and  implementation  for 
eCPD  is  that  positive  and  productive  strategies  necessary  to  improve  practice  may  be  overlooked  or 
discounted  in  efforts  to  use  technologies  to  stimulate  naive  and  unproven  ideals  of  online  communities  of 
practice  and  reflective  practice  (author  et  ak,  forthcoming).  Seen  from  a dialectical  worldview  therefore,  the 
reflexivity  of  humans  across  social  structures  suggests  that  professional  e-learning  that  produces  changes  is 
better  termed  structural  constructivism  - a reflexive  bundle  of  knowing,  doing,  thinking  and  reflecting  that 
produces  practical  knowledge  as  the  power  to  change  when  immersed  in  the  dynamism  of  practice  across 
social  structures  - enabling  critical  design  implications  for  eCPD  pedagogical  models. 


3.  PROFESSIONAL  LEARNING  AS  A STRUCTURAL 
CONSTRUCTIVIST  PROCESS:  IMPLICATIONS 

3.1  Increase  the  Capacity  for  Agency 

Approaches  for  the  design  of  pedagogical  models  for  professional  learning  to  improve  impact  on  practice, 
drawing  on  different  learning  theories,  compete  actively  in  e-learning  research  and  practices  today 
(McPherson  et  ak,  2008).  To  move  beyond  the  socially-thin  assumptions  of  the  traditional  input-output 
perspectives  on  eCPD  design  toward  an  understanding  of  eCPD  as  a change  process,  a critical  sociological 
practice  perspective  allows  educators  to  articulate  a post-structural  logic  for  learning  and  practice  change 
with  technologies,  where  professionals  have  the  energy  to  reflect  on  their  behavior  and  modify  their  actions 
as  they  seek  to  maintain  and  improve  their  social  position,  thus  making  their  reaction  to  any  ‘learning’ 
delivered  through  programmes  less  mechanical  and  more  emergent. 

In  essence,  acknowledging  the  diversity  of  social  actors  in  any  process  of  eCPD  for  changing  practice 
highlights  that  implementing  technologies  and  implementing  evidence  to  improve  practice  brings  up  issues  of 
conflict  and  confrontation  because  of  the  agency  of  all  practitioners. 

Agency  is  an  important  factor  in  the  construction  of  an  ‘authentic’  online  teaching-learning  context  to 
make  eCPD  more  effective.  It  means  educators  can  take  less  for  granted  in  terms  of  how  their  programmes 
affect  the  empowerment  of  professional  e-learners.  It  also  means  managers  and  leaders  need  to  work  with 
educators  urgently  and  take  productive  strategies  to  design  for  changing  practice  that  focus  on  new  forms  of 
controlling  teams  and  new  incentives  for  digitally  mediated  e-learning  that  is  not  distinct  or  separate  from 
their  self-directed  learning  in  the  dynamism  of  practice. 

In  today’s  digitally  mediated  and  technology-saturated  organizations,  the  dynamism  of  practice  is  also 
driven  by  the  impact  of  learning  and  other  Web  2.0  technologies  being  implemented  into  the  ‘learning- 
practice  change’  process  during  eCPD.  Research  into  the  use  of  blogs,  wikis,  podcasts,  and  asynchronous 
discussion  forums  shows  that  the  different  levels  of  agency  afforded  by  particular  social  networking 
technologies  will  affect  the  usefulness  and  usability  of  these  technologies  for  professional  learning  networks 
to  adopt  for  interaction  and  collaborative  learning  (Boulous  et  ak,  2006;  Sandars  e ak,  2007). 

3.2  Restructure  the  Structures  of  Practice 

If  the  capacity  for  agency  to  change  practice  where  professionals  work  is  curtailed  by  the  existing  social 
structures,  these  realities  make  a difference  to  what  can  be  achieved  educationally  with  eCPD  programmes. 
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eCPD  designers  have  often  used  metaphors  of  ‘learning  communities’,  ‘networks  of  practice’  or  ‘small 
groups’  to  describe  social  structures  through  which  programmes  and  technologies  are  embedded  to  support 
professionals  to  learn  and  improve  their  practices.  However,  an  absent  question  is:  Do  these  structures 
increase  the  capacity  for  agency  to  make  changes?  From  a practice  perspective,  professional  e -learners  are 
not  passive  subjects  into  which  learning  can  be  inputted,  they  are  active  social  agents  who  reproduce  and  are 
produced  by  social  structures  as  they  go  about  their  daily  working  lives  (Bourdieu  1977).  Since  their  capacity 
for  agency  is  structurally  and  digitally  mediated,  a second  implication  is  that  eCPD  pedagogical  models  need 
to  deliver  programmes  and  design  processes  to  support  healthcare  professionals  learn  about  and  modify  the 
structures  of  their  practice  so  as  to  increase  their  capacity  for  agency.  This  requires  that  eCPD  pedagogical 
models  construct  new  informal  social  stmctures  that  are  safe  and  enabling,  with  the  intention  to  channel  the 
agency  of  practitioners  that  already  exist,  and  to  allow  their  ‘social  energy’  (Bourdieu,  1980)  to  connect  and 
flow,  with  technologies  and  artefacts  that  are  designed  with  the  involvement  and  input  of  the  networks  of 
professionals  they  are  intended  for. 

4.  CONCLUSION,  RECOMMENDATIONS  & EUTURE  WORK 

To  take  one  major  professional  group,  the  medical  and  healthcare  sector,  the  evidence  on  the  low  impact  of 
eCPD  in  a recent  critical  literature  review  on  improving  impact  on  practice  (author  et  al.,  forthcoming) 
showed  that  existing  studies  consists  largely  of  self-reported  behaviour  change  from  healthcare  professionals 
who  access  stand-alone  one-off  modules,  and  based  on  experimental  study  designs.  Similar  stories  of  low 
impact  are  heard  in  other  professions  such  as  teacher  eCPD.  Self-reported  claims  of  change  are  characteristic 
of  the  positivist  epistemology  and  the  technical-rational  industrial  logic  of  ‘evidence-based’  professional 
education  and  research  that  dominates  today,  based  on  linear  causal  assumptions  of  change  that  treat  e- 
learning  as  a treatment  to  be  injected  into  professionals  to  magically  make  them  improve.  These  assumptions, 
based  on  a teleological  worldview,  have  resulted  in  the  dominance  of  behaviouralist  approaches  in  the  current 
eCPD  field. 

The  intention  of  rethinking  e-learning  philosophically  here  is  not  to  disprove  the  need  for  evidence  of 
change,  but  rather  to  note  the  way  in  which  the  rhetoric  of  the  current  normative  evidence-base  practice 
approach  positions  linear  causality  as  the  way  to  produce  impact  and  solve  the  problems  of  improving 
practice.  In  this  account,  pedagogies  are  causal  solutions  to  the  problems  of  changing  behaviour.  Because 
evaluators’  criteria  are  behaviouralist  alone  such  as  reaction  and  satisfaction,  they  make  no  critical  analysis 
and  produce  no  evidence  of  the  complexity  of  the  process  of  changing  practice  or  the  social  relationships 
involved.  Nor  do  they  require  or  support  professionals  to  make  sense  of  complexity  and  become  smarter 
about  knowing  what  they  are  getting  out  of  eCPD  with  effective  self-evaluation  tools  customized  for 
measuring  e-learning  process  impact.  Neither  have  they  rethought  their  epistemological  and  ontological 
assumptions  to  advance  theory  building  and  practice  in  e-learning. 

Reacting  to  this  lack  of  prior  research  and  the  reification  of  eCPD,  this  paper  offers  a socially  critical 
practice-based  perspective  for  knowing  and  handling  professional  e-learning  to  change  practice.  This 
epistemology  reflects  the  situated,  emergent,  and  recurrent  aspects  that  make  learning  reflexive,  and  enables 
it  to  be  understood  and  generalized  as  a structural  constructivist  longitudinal  process. 

A structural  constructivist  epistemology  of  professional  e-learning  improves  upon  existing  dominant 
cognitive  psychological  and  social  constructivist  perspectives.  By  positioning  eCPD  and  change  at  the 
structural  level  of  society,  it  raises  to  the  fore  the  two  important  implications  discussed  above  for  the 
construction  of  dynamic  teaching-learning  contexts  that  designers  need  to  be  mindful  of.  Moving  forward, 
these  features  are  necessary  to  design  e-learning  that  can  achieve  a critical  grasp  on  understanding  change 
processes  at  multiple  levels  and  with  the  various  components  of  eCPD  programmes,  in  particular  the 
affordance  of  particular  technologies  and  pedagogies.  Overall,  this  epistemology  bring  eCPD  closer  to  the 
reality  of  the  dynamism  and  contingency  of  contemporary  professional  practice,  which  is  characterized  less 
by  traditional  tightly  knit  communities  than  by  weak  and  loose  ties,  competing  objectives  and  intensional 
networks,  in  a 2F'  century  world  of  global  digitally-mediated  work  (author  et  al.,  2012). 

Given  the  issues  of  conflict  and  confrontation  during  curriculum  development  and  programme  delivery, 
as  well  as  technology  implementation  in  organizations,  can  educators  working  online  to  deliver  eCPD 
programmes  continue  to  take  for  granted  the  resources  available  to  them  and  their  students  to  produce 
change?  Even  if  they  may  have  the  intention  to  change,  professionals’  ability  to  change  practice  can  be 
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curtailed  by  the  existing  social  structures.  These  realities  make  a difference  to  what  can  be  achieved 
educationally  with  eCPD  programmes  that  a structural  constructivist  lens  grasps  conceptually  to  transcend 
socially  thin  dualistic  thinking  that  separates  learning,  change  and  practice. 

By  turning  away  from  social  constructivist  perspectives,  this  position  paper  recommends  from  a structural 
construvist  lens  on  professional  e-learning  that  an  eCPD  strategy  should  be  viewed  as  an  emergent  process  of 
formal  and  informal  practices  designed  to  achieve  an  understanding  among  management,  educators  and  users 
regarding  the  effectiveness  of  a particular  programme  and  approach  with  learning  technologies  on  improving 
impact.  Because  the  use  of  technologies  modifies  the  level  and  importance  of  agency,  it  also  recommends 
that  explanations  for  the  impact  of  eCPD  on  how  and  why  practice  changes  now  develop  tools  for  measuring 
the  agency  of  professionals  in  relation  to  what  resources  eCPD  programmes  offer.  Making  professionals 
conscious  of  their  agency  to  change  by  not  only  having  a recourse  to  subjective  reflection  on  their  behaviours 
individually  or  collectively,  such  tools  will  need  to  capture  objectively  how  their  social  interactions  and 
collaboration  with  their  teams/networks  shapes  their  agency.  This  becomes  vital  to  manage  team  reflexivity 
and  improve  performance  in  a financially  challenging  environment,  where  e-learning  is  often  touted  as  a 
cost-effective  alternative  for  CPD.  Yet  little  research  exists  on  evaluating  the  value  of  time  spent  in  and  on 
online  process,  and  is  necessary  for  making  e-learning  productive  and  empowering. 

Drawing  on  a structural  constructivist  epistemology,  future  work  will  sketch  out  a theoretical  scaffolding 
and  conceptual  model  for  designing  eCPD  as  a strategy  to  improve  impact.  This  position  paper  is  offered  to 
stimulate  debate  among  funders,  policymakers,  researchers,  designers  and  practitioners  of  professional  e- 
learning,  whichever  industry  we  might  work  in. 
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ABSTRACT 

This  paper  proposes  a novel  e-learning  system  for  technical  experiments  involving  the  construction  of  practical  electronic 
circuits;  this  system  would  meet  the  various  demands  of  individual  experimenters.  This  mixed  mode  is  beneficial  for 
practical  use  in  that  an  experimenter  who  does  not  have  sufficient  circuit  components  for  circuit  making  can  still  learn  to 
construct  practical,  large-scale  circuits  and  evaluate  their  behavior  using  virtual  measurements  and  a simulation  technique 
involving  circuit  translation  into  SPICE.  The  proposed  system  is  expected  to  be  applicable  to  various  educational  media 
(e.g.,  e-Learning,  mobile  learning,  virtual  laboratories).  The  proposed  system  is  constructed  by  combining  real  and  virtual 
circuit  making  systems,  and  performs  the  improved  circuit  recognition  of  images  including  both  real  and  virtual  circuit 
regions.  The  usefulness  and  effectiveness  of  the  proposed  system  were  evaluated  by  analyzing  circuits  made  by  ten 
university  students  in  an  actual  class. 


KEYWORDS 

e-Learning  system  for  experiments,  electronic  circuit  making,  circuit  translation  into  SPICE 


1.  INTRODUCTION 

Teaching  and  learning  the  design  and  construction  of  electronic  circuits  is  important  in  the  field  of 
technology  education.  Recently,  several  education  support  systems  have  been  developed  to  improve  students’ 
understanding  of  electronic  circuits  and  circuit-making  ability.  The  stand-alone  simulation  systems 
(Abramovitz,  2011;  Fitch,  2011)  and  virtual  laboratories  (Gurkan,  2008;  Oliver,  2009)  are  useful  for 
students  lacking  experimental  facilities.  However,  these  conventional  systems  cannot  cope  with  the  wide 
variety  of  circuits  constructed  by  individual  experimenters  because  they  are  based  on  all-purpose  or  ready- 
made learning  tools  and  apply  only  to  specific  circuits  within  a subject  area. 

To  overcome  these  disadvantages,  a distance  learning  system  has  been  developed  in  the  present  study  for 
experiments  involving  both  real  circuit  making  (RCM)  and  virtual  circuit  making  (VCM)  (Takemura,  2012). 
An  experimenter  who  has  the  required  circuit  components  but  lacks  measurement  equipment  can  learn 
circuit  making  and  analyze  circuit  behaviors  using  the  virtual  measurement  technique  of  the  RCM 
system.  Distance  learning  using  the  VCM  system  is  useful  and  effective  for  users  lacking  both  circuit 
components  and  measurement  equipment  for  making  and  analyzing  real  circuits.  Both  RCM  and 
VCM  systems  are  applicable  to  various  educational  media  (e.g.,  virtual  laboratories  and  e-Learning). 
However,  there  is  a need  for  systems  utilizing  which  users  can  learn  to  construct  larger-scale  circuits  and 
various  practical  circuits.  To  deal  with  the  shortcomings  of  the  stand-alone  systems  mentioned  previously, 
this  study  proposes  a novel  learning  mode  that  combines  RCM  and  VCM.  This  proposed  mixed  mode 
performs  the  improved  circuit  recognition  of  images  including  both  real  and  virtual  circuit  regions.  The 
mixed  mode  enables  an  experimenter  who  lacks  the  necessary  circuit  components  and  measurement 
equipment  to  learn  the  construction  of  practical  and  large-scale  circuits  and  evaluate  circuit  behavior. 
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2.  METHODOLOGY 

Figure  1 illustrates  the  proposed  e-Learning  system  for  experiments  involving  the  design  and  construction  of 
practical,  large-scale  circuits.  This  system  consists  of  individual  users’  computers  and  a remote  analysis 
system.  Individual  users  can  choose  a preferred  mode  from  three  learning  modes  (VCM,  RCM,  and  the 
mixed  mode)  depending  on  the  required  purpose  or  environment  (subsections  2. 1-2.3).  To  support  circuit 
design  and  construction,  each  student  uses  a computer  to  transmit  the  circuit  image  to  an  analysis  system  that 
runs  on  the  internet.  The  analysis  system  recognizes  the  circuit  construction  using  image  processing 
techniques  and  translates  the  structure  of  the  circuit  into  a general  circuit  description  language  (SPICE) 
(Rabaey,  2012).  This  SPICE  translation  technique  indicates  the  working  of  the  circuit  and  enables  virtual 
measurement  without  real  measurement  equipment,  and  it  can  identify  the  incorrect  components  present  in 
the  circuit  (Takemura,  2011).  This  is  an  important  step  that  aims  to  improve  the  efficiency  and  prevent  the 
occurrence  of  serious  accidents  such  as  electric  shocks  or  fire. 


Figure  1 . Schematic  of  the  proposed  e-Leaming  system  for  experiments  involving  circuit  making. 


2.1  VCM  Mode 

The  VCM  mode  of  the  system  is  useful  for  students  lacking  circuit  components,  measurement  instruments,  or 
facilities  (e.g.,  laboratories).  The  VCM  mode  (Takemura,  2012)  enables  individual  experimenters  to  use 
graphics  editors,  which  are  installed  on  a user’s  computer.  Therefore,  this  system  does  not  require  the  use  of 
proprietary  graphics  software.  Each  experimenter  can  download  virtual  circuit  components  and  a template 
circuit-board  image  from  the  analysis  system.  The  connections  of  circuit  components  on  a virtual  circuit  can 
be  indicated  by  placing  the  virtual  circuit  components  and  the  characters  representing  their  parameters  (e.g., 
IC  resistance  and  capacitance)  on  the  template  circuit-board  (breadboard)  image  using  the  experimenter’s 
preferred  graphics  editor.  The  experimenter  indicates  the  virtual-circuit  connections  by  drawing  colored  lines 
on  the  template  image  using  a graphical  user  interface  (GUI)  and  saves  the  image  of  the  virtual  circuit  in  a 
commonly  used  file  format  (e.g.,  JPEG  and  BMP).  The  virtual  circuit  image  is  transmitted  to  the  analysis 
system,  and  the  analysis  system  performs  circuit  recognition  using  image  processing  techniques  (pattern 
matching)  by  comparing  the  components  in  the  virtual-circuit  image  with  each  circuit  component  available  in 
the  server  database  of  the  analysis  system.  The  analysis  system  translates  the  virtual  circuit  into  SPICE 
without  errors  because  circuit  recognition  is  based  on  the  input  data  of  the  virtually  constructed  circuit. 
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Figure  2.  Circuit  components  for  the  VCM  mode:  (a)-(f)  virtual  circuit  components  prepared  for  connection  on  the 
circuit  hoard  image  (resistor,  potentiometer,  capacitor,  electrolytic  capacitor,  diode,  and  IC,  respectively),  (g)  9V  battery, 

and  (h)  circuit  board  image  used  to  construct  a virtual  circuit. 


2.2  RCM  Mode 

An  experimenter  having  the  circuit  components  necessary  for  circuit  making  can  learn  to  construct  a 
circuit  and  analyze  its  behavior  using  the  RCM  virtual  measurement  mode  (Takemura,  2012).  The 
experimenter  transmits  the  image  of  the  constructed  circuit  to  the  analysis  system.  The  analysis  system 
performs  pattern  matching  between  the  virtually  constructed  circuit  (drawn  using  the  VCM  system)  and  the 
circuit  constructed  using  the  RCM  mode.  This  enables  the  analysis  system  to  automatically  differentiate  the 
layout  of  each  device  and  determine  the  circuit  wiring.  In  addition,  the  analysis  system  performs  circuit 
recognition  on  the  basis  of  the  training  data  obtained  from  this  approximate  differentiation  and  the  circuit 
device  database.  The  pattern  matching  process  between  the  virtual  circuit  and  the  real  circuit  improves  the 
accuracy  of  circuit  recognition  and  SPICE  translation  and  decreases  the  computational  cost. 

2.3  Mixed  Mode 

The  mixed  mode  of  the  proposed  e-Learning  system  for  practical  circuit  making  can  translate  a circuit 
image  consisting  of  both  real  and  virtual  circuit  components.  This  mode  is  useful  for  experimenters  who 
do  not  have  sufficient  real  circuit  components  needed  to  make  a complete  circuit  because  it  enables 
them  to  study  the  construction  of  practical  and  large-scale  circuits  through  the  following  steps  (A)-(E): 

(A)  An  experimenter  designs  and  constructs  a virtual  circuit  using  virtual  circuit  components 
downloaded  from  the  analysis  system  and  saves  an  image  of  the  VCM  circuit. 

(B)  The  experimenter  constructs  a partially  real  circuit  using  real  circuit  components  and  saves  an  image  of 
the  incomplete  RCM  circuit. 

(C)  Using  a graphics  editor,  the  circuit  image  is  completed  by  placing  the  VCM  components  on  the 
incomplete  RCM  image. 

(D)  The  analysis  system  automatically  indicates  the  incorrect  components  on  the  circuit  image  on  the  basis 
of  the  difference  between  the  translated  SPICE  information  obtained  from  the  images  saved  in  (A)  and 
(C). 

(E)  The  experimenter  can  study  the  behavior  of  the  circuit  on  the  basis  of  SPICE  information.  Measurement 
instruments  are  not  needed  to  check  circuit  behavior. 

In  this  study,  the  image  analysis  technique  for  circuit  recognition  was  improved  to  make  it  usable  for 
circuit  images  including  both  real  and  virtual  circuit  regions.  Concretely,  the  database  of  the  circuit 
components  for  pattern  matching  techniques  was  increased  to  cover  both  RCM  and  VCM  modes,  and  a 
morphological  operation  (Haralick,  1987)  is  used  to  detect  the  connected  position  of  each  component  on  the 
breadboard  and  adjust  the  size  of  the  discriminated  virtual  components  on  the  basis  of  the  size  of  real 
components. 
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2.4  Experimental  Methodology 

The  proposed  system  was  evaluated  by  analyzing  the  circuits  designed  and  constructed  by  ten  experimenters 
in  an  actual  class  at  Tokyo  University  of  Agriculture  and  Technology.  Figure  3(a)  shows  the  diagram  of  a 
practical  sound  processing  circuit  for  electric  guitars  to  be  constructed  using  the  mixed-mode  of  the  proposed 
system.  The  physical  circuit  components  to  be  used  to  construct  a simple  circuit  (an  active  filter)  (Fig.  3(b)) 
were  provided  to  each  student  on  the  assumption  that  experimenters  who  learn  circuit  construction  using 
an  e-Learning  system  do  not  have  sufficient  circuit  components.  Therefore,  the  circuit  components 
provided  to  the  students  were  not  sufficient  to  construct  the  circuit  shown  in  Fig.  3(a),  and  thus,  the 
mixed  mode  of  the  proposed  system  is  indispensable  to  the  construction  of  the  complete  circuit. 


Figure  3.  Circuit  diagrams  to  evaluate  the  proposed  system:  (a)  circuit  to  be  constructed  using  the  mixed  mode  of  the 
proposed  system  and  (b)  circuit  to  be  constructed  by  individual  users  using  the  available  circuit  components. 


3.  RESULTS  AND  DISCUSSION 


Figure  4(a)  and  (b)  show  examples  of  the  circuit  designed  using  the  VMC  mode  and  the  circuit  constructed 
using  the  proposed  mixed  mode  by  an  experimenter,  respectively.  Figure  4(c)  shows  an  example  of  the 
simulation  results  (sound  waves  of  the  input  and  output  signals)  obtained  from  the  SPICE  translation  of  the 
circuit  constructed  using  the  mixed  mode.  The  analysis  system  provided  simulated  the  sounds  and  provided 
the  simulation  results,  enabling  the  experimenter  to  examine  the  behavior  of  the  circuit  and  the  effect  of 
sound  processing  (synthesized  sound)  on  the  basis  of  the  virtual  circuit. 


000 
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Figure  4.  Results  of  circuit  construction  using  the  mixed  mode:  (a)  circuit  designed  using  the  VCM  mode,  (b)  image  of 
the  circuit  constructed  using  the  proposed  mixed  mode;  (c)  SPICE  simulation  based  on  circuit  translation(sound  waves  of 

the  input  and  output  signals). 
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The  proposed  system  was  evaluated  by  analyzing  the  circuits  constructed  by  ten  students  at  Tokyo 
University  of  Agriculture  and  Technology.  The  following  positive  responses,  which  pertain  to  the  usefulness 
and  efficiency  of  the  proposed  system,  were  obtained  from  all  students: 

• The  mixed  mode  of  the  systems  is  useful  because  it  enables  e-Learning  of  topics  such  as  circuit  design 
and  experiments  involving  practical  circuit  making. 

• The  proposed  system  is  instructive,  and  individual  students  can  study  practical  circuits.  The 
system  enables  enlargement  and  optimization  of  a physically  constructed  circuit  using  the 
mixed  mode  of  the  e-Learning  system. 

However,  there  were  also  a few  technical  disadvantages  and  suggestions  for  improvement: 

• A remote  educational  system  to  study  digital  circuits  is  expected. 

• The  size  of  the  experimenter’s  system  should  be  reduced. 


4.  CONCLUSION 

This  paper  deals  with  a novel  e-learning  system  for  technical  experiments  involving  the  construction  of  large- 
scale,  practical  electronic  circuits.  The  proposed  mixed-mode  system  is  developed  by  combining  RCM  and 
VCM  systems.  The  usefulness  and  effectiveness  of  the  system  were  verified  by  analyzing  electronic  circuits 
constructed  by  ten  students  in  an  actual  university  class.  Positive  responses,  which  pertain  to  the  usefulness 
and  efficiency  of  the  proposed  system,  were  obtained  from  all  the  students.  The  following  steps  are  necessary 
to  practically  realize  the  proposed  system: 

• A mobile  communication  tool  instead  of  an  experimenter’s  computer  is  expected  to  reduce  the  size  of 
the  system. 

• Computational  costs  for  circuit  recognition  procedure  should  be  decreased. 
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ABSTRACT 

The  paper  describes  component-based  approach  (CBA)  for  learning  management  system  development.  Learning  object  as 
components  of  e-learning  courses  and  their  metadata  is  considered.  The  architecture  of  learning  management  system 
based  on  CBA  being  developed  in  Riga  Technical  University,  namely  its  architecture,  elements  and  possibilities  are 
described.  Modern  technologies  used  for  implementation  of  the  systems  are  outlined. 


KEYWORDS 

Component-based  approach,  learning  management  system,  technologies 


1.  INTRODUCTION 

The  problem  of  providing  a high  quality  learning  and  teaehing  proeess  is  beeoming  more  aetual  sinee 
distanee  edueation,  open  edueation  and  life-long  learning  are  very  popular  and  useful  forms  of  enhaneing 
edueation.  The  availability  of  the  new  software  teehnologies  possibilities  opens  the  doors  to  everyone  who 
wants  to  be  involved  in  the  proeess  of  developing  e-eourses.  Many  seientists  work  on  the  teehnologies  and 
methods  of  adaptive  learning  [Oppermann  R.  et  al,  1997;  Wey  Chen  J.,  2010;  Zaitseva  L.,  Buie  C.,  2011], 
models  [Buie  J.,  Zaitseva  L.,  2007;  Tsinakos  A.,  2010;  Heineman  G.T.,  Couneill  W.T.,  2001;  Seorm,  2013] 
and  standards  [LOM,  2002.;  LTSA,  2003].  Different  learning  management  systems  and  portals  [eCollege, 
2013;  Edmodo,  2013;  Moodle,  2013;  WeBWork,  2013],  speeialized  and  universal  learning  systems 
[Coursera,  2013;  eFront,  2013;  Edutools,  2013]  are  being  ereated  and  broadly  used.  Considering  sueh  a 
variety  of  developed  and  a huge  open-soureing  it  is  eonvenient  and  appropriate  to  utilize  a reusability  feature, 
i.e.,  apply  eomponent-based  approaeh  of  software  engineering  to  an  e-eourse  development. 

The  aim  of  the  paper  is  to  deseribe  a learning  management  system  development  proeess  taking  into 
aeeount  eomponent-based  approaeh  prineiples.  The  following  seetion  shows  learning  objeets  as  the  main 
eomponents  of  an  e-eourse.  The  third  ehapter  reviews  learning  management  system  arehiteeture.  The  last 
ehapter  outlines  the  teehnologies  used  to  implement  the  deseribed  system. 


2.  COMPONENT-BASED  APPROACH  TO  SOETWARE  DEVELOPMENT 

Component-based  engineering  [Fleineman  G.T.,  Couneill  W.T.,  2001]  is  one  of  the  trends  of  the  software 
engineering.  This  approaeh  supposes  that  software  systems  are  assembled  from  reusable  eomponents  (RC). 
These  eomponents  are  developed  earlier  and  saved  in  a repository  of  RC.  Then  they  are  used  for  new 
system’s  development.  Component-based  approaeh  (CBA)  also  ean  be  sueeessfully  applied  for  learning 
management  system  (LMS)  and  e-leaming  eourses  as  basie  elements  of  LMS  development. 

E-leaming  eourse  is  a set  of  learning  objeets  (LO)  implemented  using  modem  eomputer  teehnologies  and 
eonsolidated  into  a seenario  that  is  ereated  to  organize  a learning  proeess  of  a definite  topie  of  a subjeet  by 
student  or  students’  group  (Collaborative  learning). 

Thus,  LOs  are  reusable  eomponents  of  an  e-leaming  eourse.  Learning  objeets  ean  be  of  two  types: 
learning  objeet  information  (LOI)  and  learning  objeet  task  (LOT).  LOI  deseribes  theoretieal  information  of  a 
eourse  - different  eoneepts,  examples,  ete.  LOT  is  used  for  knowledge  eontrol  and/or  training  during 
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acquisition  of  a course.  LO  has  a complex  structure  and  can  be  not  just  of  various  types  (LOI  and  LOT),  but 
also  of  different  kinds  and  parts  within  a type.  Both  for  LOI  and  LOT  the  kinds  are  the  same  since  a LOT  is 
supposed  to  be  used  to  evaluate  knowledge  and/or  skill  level  on  a definite  LOT  So,  the  kinds  are  as  follows: 
structure,  definition,  example  and  rule  [Zaitseva  L.V.  et  al,  1989],  - depending  on  what  exactly  is  being 
described  in  a LOT  As  well  there  can  be  various  parts  for  each  LO: 

• LOI  - main,  example,  explanation.  Example  and  explanation  are  optional  LOIs  of  different  detailing 
level; 

• LOT  - two  parts:  general  task  and  individual  task.  The  general  task  is  the  same  for  all  students  and 
individual  task  is  generated  for  each  student  separately  to  specify  a general  task.  Every  LOT  according  to 
a type  has  one  or  several  answers,  which  are  also  referred  as  separate  LO  - LOA.  As  well  there  can  be 
included  comments  for  LOTs  in  general  and  every  LOA  particularly  - LOC  as  well  of  different  detailing 
level. 

In  addition  to  LO’s  complexity  there  is  a list  of  metadata  that  should  be  considered  while  developing  an 
e-course.  The  full  list  of  LOs  metadata  is  available  in  IEEE  standard  [LOM,  2002].  It  is  evident  that  all  the 
data  is  required  to  implement  an  e-course  of  high  quality,  but  there  is  always  more  critical  data  that  should  be 
taken  into  account  for  different  purposes  (Table  1). 


Table  1 . LOs  metadata 


LO  metadata 

Purpose 

Description 

LO  title,  description,  language,  platform, 
keyword,  coverage/topic,  type  (LOI  - 
main,  example,  explanation),  specialty, 
program  (study  level  - bachelor,  master, 
college,  etc.),  purpose,  copyright, 
restrictions 

The  initial  list  of  reusable  LOs 
to  be  included  into  an  e-course 

The  main  criteria  to  choose 
from  the  list  of  LOs  available 
in  repository 

LO  significance,  mode  of  using  (control, 
training),  number  of  tries  (LOT),  time 
(acquisition  - LOI  or  performing  - 
LOT),  type  (LOI  - main,  example, 
explanation),  difficulty 

Possible  minor  modifications  to 
adapt  for  an  e-course 

Means  just  adaptation  of  the 
parameters,  not  exactly 

modifications  of  LO  itself 

LO  representation  (media),  LOT’S  type 
(multiple  choice,  word,  number,  etc. 
[Zaitseva  L.  et  al,  2005]),  language 

Possible  major  modifications  to 
adapt  for  an  e-course 

It  is  consider  as  a major 
modifications,  if  changing  of 
media  or  translation  is 
involved 

LOI  kind  (definition,  example,  rale, 
structure),  purpose,  topic,  LOT’s  type 

Decision  on  developing  new 
LOs 

In  the  case  when  there  are  no 
LOs  of  specific  kind  or  on 
definite  topic  of  a course,  or 
there  are  just  one-type  LOTs 

As  it  was  mentioned  earlier,  LOs  are  components  of  an  e-course  side  by  side  with  different  processing 
programs/modules.  Thus,  to  apply  component-based  approach  for  LMS  and  e-leaming  course  (ELC) 
development  repositories  of  learning  objects  and  program  components  are  necessary. 


3.  LEARNING  MANAGEMENT  SYSTEM  ARCHTECTURE 

Different  classes  of  users  can  use  LMS.  They  are:  student,  tutor,  e-leaming  courses  author  (ELCA), 
administrator,  operator,  but  basic  classes  of  users  are  first  three.  That’s  why  to  present  the  architecture  of 
LMS  (Fig.  1)  we  selected  only  three  agents.  LMS  also  has  a main  program  module  (kernel  of  a system)  that 
provides  users  identification  and  connection  to  specified  agent,  a unified  interface  for  all  agents,  organization 
and  maintaining  of  connection  session,  repository  and  databases  maintaining,  etc. 
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Figure  1.  Learning  management  system  architecture 

Learning  objects  and  their  metadata  are  saved  in  LO’s  repository  and  can  be  used  by  ELC  author  to 
develop  e-leaming  course.  Developed  and  tested  ELC  is  saved  in  e-course’s  database.  A tutor  after  reviewing 
offers  it  to  students  and/or  groups  of  students.  Students  can  use  e-leaming  course  for  learning,  teaching, 
knowledge  control  or  training.  Student’s  and  group’s  database  includes  information  about  students  and 
groups  of  students,  as  well  as  a protocol  describing  a work  with  e-course. 

The  main  functions  provided  by  ELCA  agent  are  e-leaming  course  development  and  testing.  A process 
of  an  e-course  development  based  on  CBA  consists  of  the  five  stages:  1)  define  a goal  and  objectives  of  the 
e-course;  2)  develop  the  e-course  scenario  using  information  about  LOs  and  their  metadata  saved  in  a 
repository.  Usually  a scenario  of  an  e-course  is  represented  as  a graph  with  LOs  as  vertexes  and  edges  show 
the  connections  types  between  them  [Zaitseva  L.,  Buie  C.,  2011];  3)  select  necessary  learning  objects  from  a 
repository  and  check  them.  An  analysis  and  reviewing  of  the  selected  learning  objects  is  performed  during 
this  stage  to  find  out  whether  they  fully  correspond  to  the  goal  and  objectives  of  the  e-course,  as  well  as  to 
make  a decision  on  each  LO:  to  use  it  with  no  modifications;  edit  it  adapting  to  the  e-course  requirements; 
develop  fully  new  LO(-s);  4)  modify  the  selected  LOs,  develop  new  LOs  and  test  them.  Modifications  can  be 
as  follows:  add  new  information  to  LO,  remove  unnecessary  information  according  to  the  e-course  author 
and/or  represent  existing  information  by  means  of  other  media;  5)  integrate  LOs  into  the  e-course  according 
to  the  scenario  developed  in  the  stage  2.  The  next  step  is  testing  of  the  e-course  in  general.  Afterwards  it  is 
placed  into  e-course’s  DB.  If  during  and  ELC  development  the  new  LOs  were  created  or  modified  the 
existing  ones,  author  can  save  them  in  a repository. 

The  function  „E-course  statistics”  allows  author  to  get  information  about  students  work  with  ELC.  After 
analyzing  it  author  can  improve  a quality  of  an  e-course  by  making  some  modifications,  for  example,  adding 
more  detailed  representation  of  learning  information  with  new  examples  or  specifying  a text  of  a task. 
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The  Tutor  agent  provides  the  following  functions: 

• E-course  review.  Before  making  an  EEC  available  for  students  tutor  has  to  test  it  and  evlauate  it’s 
correspondance  to  goals  and  tasks  of  a subject  taught  as  well  to  make  a decision  on  expoliting  it  without 
any  modifications  or  make  some  amendments,  for  example,  add  more  examples,  questions  and/or  tasks. 
In  the  last  case  a copy  of  an  e-course  with  tutor  made  changes  is  saved  in  e-course’s  DB. 

• Mode  definition.  Depending  on  a chosen  strategy  of  an  EEC  using  [Zaitseva  L.,  Buie  C.,  2011]  tutor  can 
set  paramateres  of  a course  exploiting  that  are  usually  specified  for  a group  of  students.  For  example,  an 
EEC  is  supposed  to  be  used  for  totally  unassisted  learning,  training  or  knowledge  control.  For  knowledge 
control  (KC)  it  is  possible  to  determine  KC  method,  a number  and  difficulty  levels  of  tasks,  etc.  [Zaitseva 
L.V.  et  al,  1989]. 

• Statistics.  This  function  allows  a tutor  the  possibility  to  get  the  detailed  information  about  student  work 
results. 

The  Student  agent  ensures  a work  in  four  modes  for  students: 

• Learning.  This  mode  gives  student  a possibility  to  learn  independently  any  course  that  is  available  in  e- 
courses’  DB,  using  Teaching,  Knowledge  control  and/or  Training. 

• Teaching.  The  mode  provides  an  e-course  teaching  that  was  assigned  by  a tutor  according  to  developed 
scenario. 

• Knowledge  control  (KC)  is  used  to  test  and  evaluate  students’  knowledge  and  skills  level.  If  the  mode  is 
assigned  by  tutor,  a student  answers  questions  and  performs  tasks  according  to  set  control  parameters.  If 
the  KC  is  chosen  by  student  to  test  own  knowledge  and  skilss  level  (self-control)  then  he/she  can  choose 
as  well  a number  of  tasks,  difficulty  level  and  detailing  of  comments. 

• Training  is  supposed  for  mastering  skills  on  tasks  solving  on  chosen  or  assigned  by  tutor  topic.  In  this 
case  it  is  the  best  practice  to  provide  detailed  comments  on  every  student  action. 

Learning  management  system  includes  also  various  supplementary  program  components  (modules)  to 

ensure  learning  and  teaching  functions,  namely:  module  for  choosing  tasks  from  similar  ones  to  ensure  that 

students  get  different  questions  during  learning  process;  module  for  generating  input  data  and  calculating 

results  while  tasks  performing  and  solving  calculation  assignments  and  others. 


4.  TECHNOLOGIES  EOR  LEARNING  MANAGEMENT  SYSTEM 
IMPLEMENTATION 

Considering  complexity  of  e-leaming  system  architecture,  it  is  implemented  based  on  a multi-layer  structure, 
which  is  standard  for  modem  business  software.  System  itself  is  based  on  web  technologies  since  it  is  the 
most  efficient  way  to  deliver  content.  As  well  they  have  developed  dramatically  recently  and  allow  simple 
processing  of  all  kinds  of  information  used  in  e-leaming  process.  Client  side  is  a browser-based  user  interface 
with  all  functionality  needed  to  show  all  e-leaming  components  for  user,  while  server  side  contains  all 
processing  logic  and  databases/repositories. 

Client  side  realized  by  using  current  web  technology  standard:  Ajax,  which  allows  dynamically  changing 
content  depending  on  input  and  reducing  processing  requirements  on  server,  and  considering  increased 
requirements  for  security.  Secure  Socket  Layer  (FITTPS  protocol)  is  a “must  have”  technology.  To  design 
user-friendly  interface  (UI)  cross-browser  integration  framework  should  be  used  - j Query  [jQuery,  2013], 
Prototype/Scriptaculous  [Prototype,  2013;  Scriptaculous,  2013],  MooTools  [MooTools,  2013],  Dojo  [Dojo, 
2013].  One  of  the  most  optimal  frameworks  for  client  side  is  MooTools,  which  ensures  such  an  UI  rendering 
that  it  looks  the  same  on  every  popular  browser  currently  on  market. 

Server  side  implements  business  logic  and  DBMS  system.  Considering  a length  of  a life-cycle  of 
software,  five  to  seven  years,  technologies  used  in  business  logic  implementation  must  be  solid,  i.e.,  have 
long  history  of  support  and  without  major  changes  in  syntax  through  version  history  and  with  a big  enough 
community  of  developers,  like  Java,  .Net,  PHP.  Therefore  server  side  logic  is  performed  on  Java,  which  has 
efficient  integrated  environments  also  there  are  a lot  of  frameworks  available.  As  well  one  of  the  advantages 
is  that  almost  everything  is  for  free,  so  development  costs  are  highly  reduced.  Taking  into  account  that  OSGi 
container  [OSGi,  2013]  is  used  for  different  modules,  the  GlassFish  Java  server  [GlassFish,  2013]  was 
chosen  for  providing  a performance  of  LMS.  OSGI  allows  separating  some  logic  and  functionality,  and  also 
dynamically  adding  new  content  to  server. 
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To  ensure  data  storage  in  repositories  DBMS  PostgreSQL  is  employed  since  it  it’s  free,  open-source, 
highly  customizable  and  has  powerful  tools  for  data  managing  and  processing  inside  database  [PostgreSQL, 
2013]. 

5.  CONCLUSION 

The  component-based  approach  to  develop  learning  management  system  allows  reducing  time  and  other 
resources  consumption.  Since  this  approach  is  based  on  the  reusable  components  using  to  create  a new 
product  it  also  facilitates  the  whole  process.  In  case  of  LMS  implementation  the  benefits  of  CBA  are  for  both 
the  system  itself  and  e-courses  that  are  developed  in  it.  Thanks  to  it  and  used  technologies  (OSGi)  existing 
modules  can  be  easily  modified  and  the  new  ones  added,  and  the  same  situation  is  with  learning  objects  from 
available  repositories  that  are  treated  as  components  for  an  e-course. 

As  further  work  we  plan  to  enhance  the  existing  LMS  by  using  modem  technologies  as  well  to  transfer 
already  developed  e-leaming  courses  into  the  improved  learning  management  system. 
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ABSTRACT 

Learning  portfolios  play  a critical  role  in  fostering  learning  and  assessing  learning  achievement.  For  easy  access  and  more 
efficient  processing,  various  electronic  portfolio  systems  have  being  developed  and  receiving  widespread  adoption. 
Contemporary  electronic  portfolio  systems  are  designed  and  developed  proprietarily,  such  that  the  limited  functionalities 
of  each  system  are  available  to  particular  users  only,  and  it  is  difficult  to  integrate  them  due  to  diverse  access  interfaces. 
However,  just  like  other  pedagogy  support  systems,  e-portfolio  systems  must  be  renovated  continuously  to  make  them 
adapt  to  the  ever-changing  learning  contexts  and  meet  diverse  requirements  from  teachers,  educators,  and  administrators. 
Besides,  to  facilitate  sharing  and  consolidating  resources  from  different  e-portfolio  systems,  it  is  of  great  importance  to 
build  easy-accessible  as  well  as  open  e-portfolio  systems.  In  consideration  of  the  necessary  agility,  versatility,  and 
accessibility,  the  present  work  proposed  the  concept  of  learning  portfolio  as  a service  and  realized  it  with  a restful  styled 
approach.  This  article  describes  the  design  rationales  and  implementation  of  the  learning  portfolio  as  a service  system, 
and  analyze  its  impact  on  educational  facets. 


KEYWORDS 

E-learning,  electronic  learning  portfolio,  cloud  computing,  SaaS,  Restful  Web  services. 


1.  INTRODUCTION 

Learning  portfolios  have  being  widely  used  to  record  all  kinds  of  relevant  data  during  the  course  of  teaching 
and  learning  activities,  and  insightful  information  that  can  improve  learning  performance  could  be  further 
distilled  from  these  records.  For  the  sake  of  efficiency  and  easy  access  in  the  age  of  Internet,  many 
Web-based  learning  platforms  take  in  e-portfolio  systems  to  fulfill  the  task  of  tracking  learning  activities  and 
documenting  the  corresponding  artifacts.  However,  to  adapt  to  ever-changing  learning  environments  and 
meet  diverse  requirements  from  students,  teachers,  administrators,  e-portfolio  systems  need  to  be  agile  and 
versatile.  Furthermore,  because  cross-institute  integration  of  portfolio  resources  enables  cooperative  analysis, 
planning,  and  management  among  different  education  institutes,  administrative  and  accreditation  agencies, 
functionalities  of  ideal  e-portfolio  systems  need  to  be  accessible. 

Just  like  other  Web-based  learning  management  and  contents  systems,  most  currently-deployed  e- 
portfolio  systems  have  two  major  drawbacks,  which  include  (1)  proprietary  design  and  implementation  make 
each  e-portfolio  system  be  available  to  limited  users  who  teach  or  learn  in  particular  environment;  (2)  non- 
common application  programming  interface  leading  to  difficulty  in  integration  of  distributed  but 
complementary  resources.  Fortunately,  the  service -based  approach,  to  certain  extent,  is  able  to  resolve  these 
shortages  (Hu  and  Chen,  2008). 

Among  many  advantages  of  service-based  pedagogical  resources,  the  most  attractive  one  to  both  content 
consumers  and  providers  is  the  open  access  method  that  are  based  on  existing  standardized  techniques 
including  HTTP,  XML,  etc.  Not  only  being  able  to  couple  distributed  and  heterogeneous  resources  easily,  the 
standardization,  the  derived  openness  and  interoperability  also  facilitate  the  access  to  service-based  resources 
and  ease  the  composition  of  them. 
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In  view  of  the  importance  of  an  agile,  versatile,  and  accessible  learning  portfolio  system,  the  present 
research  work  aimed  to  develop  a system  that  realizes  the  concept  of  "learning  portfolio  as  a service" 
(LEPORAAS)  according  to  the  restful  style.  Besides  the  fundamental  functionality  of  a typical  e-portfolio 
system,  the  LEPORAAS  will  enable  prompt  assessment  of  learning,  pervasive  assessment  in  diverse 
instructional  and  learning  situations,  efficient  cross-institute  cooperation,  and  macro  administrative  works 
and  decision  making. 


2.  PRIOR  RESEARCH  WORKS  AND  RELATED  TECHNIQUES 


2.1  Requirements  of  an  Ideal  E-Portfolio  System 

In  general,  education  institutes  use  portfolios  to  document  students'  learning  activities,  progress,  and  concrete 
artifacts.  The  original  and  main  purpose  of  learning  portfolios  is  providing  educators  and  administrators  an 
alternative  tool  to  assess  students'  learning  (Ryan,  2011,  Cavalier,  2011)  from  not  only  result-oriented,  but 
process-oriented  viewpoints.  Portfolios  support  summative  and  formative  assessment  (Cans,  2009),  which 
usually  need  reflection  of  artifacts  that  were  developed  during  the  course  of  learning  and  thus  is  more 
difficult  to  conduct  than  conducting  quantitative  and  summative  assessments,  such  as  standardized  tests.  The 
electronic  version  of  learning  portfolios  offer  extra  merits  including  storage  of  multimedia  contents,  easy 
access,  fast  index  and  search,  efficient  and  precise  analysis  in  depth. 

Beyond  its  typical  application  to  assessment,  functionality  of  e-portfolios  has  been  proved  to  be  multi- 
facet. Many  prior  research  works  indicated  that,  main  effects  of  portfolio  also  include  improving  students' 
knowledge  and  understanding,  creating  greater  self-awareness  and  encouragement  to  reflection  (Inacio  and 
Salema,  2011),  and  nourishing  the  ability  to  learn  independently  (Buckley  et  ah,  2009,  Hays,  2004,  Zembal- 
Saul  et  ah,  2002).  Additionally,  along  with  Websites  and  Eacebook,  e-portfolios  have  been  applied  to  aid  pre- 
service teachers  in  understanding  how  to  integrate  technology  into  their  instruction  (Shaltry  et  ah,  2013).  The 
study  conducted  by  Lambe  et  al.  shows  that  e-portfolio  can  assess  aspects  of  pre-service  teacher  education 
program  within  the  specific  context  of  special  needs  education,  and  can  demonstrate  a range  of  teaching 
competencies  (2013).  E-portfolio  also  can  serve  as  an  effective  tool  to  facilitate  the  communications  among 
students,  teachers,  and  parents  who  can  keep  track  of  children's  classroom  learning  through  the 
corresponding  e-portfolios  (McLeod  and  Vasinda,  2009).  Not  surprisingly,  career  placement  could  be  more 
suitable  and  precise  by  referencing  long-term  portfolios  (Brennan  and  Lennie,  2010)  of  job  seekers,  rather 
than  relying  on  the  result  of  a single  placement  test.  The  reason  is  that  contents  in  e-portfolio  reflects  a 
student's  strength,  weakness,  interests,  and  particular  the  progress  in  specific  domains. 

In  recent  years,  ICTs  facilitate  new  learning  approaches  such  as  collaborative  learning  and  edutainment. 
In  addition,  situations  for  learning  have  been  broadened  by  wireless  and  broadband  communication  networks, 
which  realized  the  concept  of  mobile  or  pervasive  learning.  In  summary,  all  new  aspects  of  teaching  and 
learning  activities  correspondingly  induce  new  requirements  for  e-portfolio  systems.  That  means  e-portfolio 
systems  need  to  be  adaptive  and  versatile,  in  order  to  meet  emergence  of  ceaseless  new  requirements.  Besides, 
they  need  to  be  accessible  for  easy  consolidation  of  portfolio  resources  from  multiple  institutes  (Ritzhaupt  et 
ah,  2008). 

In  addition  to  the  aforementioned  aspects,  the  widespread  necessity  for  e-portfolio  systems  by  institutes 
leads  to  two  more  requirements  of  an  ideal  e-portfolio  system:  a cost-efficient  delivery  mode  and  an  elastic 
usage/payment  scheme.  The  cost-efficient  delivery  mode  reduces  burden  on  preparing  and  maintaining 
software,  while  the  elastic  payment  scheme  makes  cost  be  proportional  to  actual  usage.  Both  of  these  two 
aspects  are  valuable  to  educational  institutes  due  to  their  great  deal  of  variety  in  demand  for  e-portfolios 
functionality  and  financial  strength.  Taking  all  of  these  advantageous  aspects  into  account  simultaneously, 
provisioning  learning  portfolios  as  a service  is  a reasonable  and  promising  option. 
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2.2  Software  as  a Service  and  Restful  Styled  Web  Service 

As  one  of  the  service  model  in  cloud  computing,  software  as  a service  (SaaS)  (Liu  et  al.,  2010,  Goth,  2008) 
should  be  the  one  which  most  end  users  of  software  can  benefit  from.  SaaS  reshaped  the  paradigm  through 
which  software  deliver  its  functions  to  users.  Unlike  traditional  software  that  need  to  pre-acquired  and 
installed  on  particular  host(s),  SaaS  deliver  user  functions  that  were  wrapped  in  Web-based  interfaces 
through  the  Internet,  or  more  specifically,  the  Web  service  techniques  (Petrie,  2009,  Yi  and  Blake,  2010). 
Because  the  corresponding  program  and  configuration  files  are  controlled  and  maintained  by  service 
providers  on  remote  sites,  users  of  SaaS  are  able  to  access  the  latest  updated  functions  of  software  without 
installation  and  maintenance  by  themselves. 

The  most  valuable  advantage  of  SaaS  is  that  SaaS  enables  a fairer  consumption/payment  scheme: 
pay-per-use  in  software  industry.  The  advantages  of  SaaS  already  attracted  volume  of  users  who  want  to  use 
software  functionality  and  pay  for  it  fairly,  but  are  reluctant  to  manage  and  maintain  software. 

When  Web  service  techniques  were  applied  in  the  provision  of  e-learning  functionalities  and  materials,  its 
platform-neutrality  characteristic  means  no  matter  which  implementation  techniques  and  operating  platforms 
were  chose,  once  an  instructional  resource  was  developed  or  just  wrapped  as  a service,  all  clients  on  the 
Internet  can  send  compliant  request  to  access  it.  Obviously,  that  will  leverage  the  reusability  and  sharing  of 
educational  resources  beyond  the  current  level. 

To  resolve  the  overhead  issue  by  the  SOAP  based  Web  service,  the  representational  state  transfer  (REST) 
architectural  style  (Fielding,  2000)  offers  a promising  approach.  The  restful  Web  applications  (Vinoski,  2008) 
emerged  as  an  lightweight  alternative  for  realizing  the  concept  of  SaaS,  which  is  one  of  the  major  service 
models  in  cloud  computing  (Dillon  et  al.,  2010)  environment.  In  a restful  Web  application,  everything  that 
could  be  accessed  or  operated  are  treated  as  resources,  and  resources  must  be  identifiable  via  an  uniform 
naming  scheme,  or  practically,  the  uniform  resource  identifier  (URI).  In  contrast  with  its  heavyweight 
counterpart:  the  SOAP-based  Web  service,  the  restful  Web  service  associates  standard  HTTP  methods  with 
operations  that  intended  to  be  performed  on  resources. 

Due  to  the  uniform  identifying  scheme  and  access  (operation)  interface,  the  restful  approach 
significantly  reduces  the  complexity  that  are  caused  by  processing  of  SOAP-encoded  messages,  such 
reduction  offers  users  and  bystanders  a lightweight  option  for  adopting  Web  services.  Moreover,  the  plain 
HTTP-based  access  interface  eases  the  integration  of  popular  and  lightweight  techniques  such  as 
asynchronous  JavaScript  and  XML  (AJAX)  into  client  side  of  restful  Web  applications,  which  further  makes 
more  service  consumers  using  devices  with  various  form  factors  to  access  online  resources  pervasively. 

In  general,  the  restful  style  have  being  applied  to  develop  Web  applications  with  diverse  purposes 
(Belimpasakis  and  Moloney,  2009,  Rissanen  et  al.,  2010,  Gao  et  al.,  2010,  Qian  and  Xianglong,  2010). 
Considering  the  advantages  of  restful  Web  applications,  this  research  work  proposed  the  learning  portfolio  as 
a service  and  realized  it  with  the  restful  style  and  the  supported  techniques.  The  implementation  work 
embodies  merits  including  lightweight,  pervasive  access,  scalability,  and  flexibility. 


3.  REALIZATION  OF  A RESTFUL  LEARNING  PORTFOLIO  SERVICE 

This  section  explains  the  design  rationales  of  the  restful  learning  portfolio  service  system,  as  well  as 
describes  how  the  design  concepts  were  realized  accordingly. 

3.1  Design  Rationales 

To  embody  the  concept  of  learning  portfolio  as  a service,  a corresponding  system  was  designed  and 
implemented.  The  system  consisting  of  major  components  is  illustrated  in  figure  1.  The  design  of  the 
interface  for  accessing  this  e-portfolios  service  system  complied  with  the  following  constraints:  first,  all  types 
of  resources  will  be  organized  in  hierarchical  style.  For  example,  a user's  portfolio  consists  several  of 
sub-portfolios  that  in  turn  comprising  different  types  of  records  and  artifacts  associated  with  a specific  user. 
Second,  identifiers  of  all  resources  follow  a common  URI  pattern,  and  thus  could  be  represented 
hierarchically.  For  example,  to  identify  a particular  (say,  with  ID  =10)  certificate  in  a sub-portfolio  recording 
a user's  all  certificates,  we  can  use  http://www.eportfolio.org/usemame/certificate/10. 
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Figure  1 . Architecture  of  The  Restful  Learning  Portfolio  Service  System 


Third,  the  four  basic  HTTP  methods:  POST,  GET,  PUT,  and  DELETE  are  uniformly  used  to  map  the 
CRUD  (Create,  Read,  Update,  Delete)  operations  on  a resource,  respectively.  Accordingly,  to  create  a new 
instance  of  the  certificate  record  for  a particular  user,  a POST  request  including  the  HTTP  verb  along  with 
the  request  contents  (input  data)  would  be  sent  to  http://www.eportfolio.org/username/certificate/new/. 

3.2  Implementation 

To  realize  the  design  concept,  the  Apache  Tomcat  was  used  to  serve  as  the  Web  and  application  server,  the 
restlet  was  used  for  frameworking  the  restful  Web  application  with  Java  servlet  and  JSP  techniques.  To 
persist  all  data  records  regarding  contents  in  users'  portfolios,  mySQL  was  the  choice.  On  the  client  side  of 
the  proof-of-concept  system,  the  AJAX  was  applied  to  send  the  four  basic  HTTP  requests,  as  well  as  to 
receive  and  handle  the  response  messages  from  the  service  provider.  Typical  restful  applications  allows  that 
resources  have  different  formats,  e.g.  plain  text,  XML,  and  JSON  (JavaScript  Object  Notation),  etc. 


4.  ADVANTAGES  AND  IMPACTS 

The  advantages  of  the  learning  portfolio  as  a service  and  the  corresponding  impact  on  multiple  facets  are 
discussed  in  this  section. 

4.1  Agility  and  Versatility 

With  adoption  of  the  SaaS  concept,  consumers  of  e-portfolio  services  can  always  use  the  latest  functionality 
but  do  not  need  to  install  and  maintain  various  software  any  more,  instead,  they  pay  for  what  they  actually 
used.  Doubtless,  this  flexibility  of  payment  scheme  will  reduce  waste  that  are  caused  by  pre-licensed  but 
under-utilized  software.  Moreover,  the  pay-per-use  scheme  enables  a finer-grained  budget  allocation. 

By  inheriting  the  most  valuable  merit  of  the  service-oriented  architecture  (SOA)  (Yau  et  al.,  2009,  Joseph, 
2006)  for  building  up  software  systems,  innovative  functionalities  of  e-portfolio  service  system  could  be 
developed  in  shorter  time  frame  by  mashing  up  available  services,  rather  than  developing  everything  from 
scratch.  An  most  importantly,  the  useful  services  could  be  within  or  outside  of  the  present  system.  This 
versatility  keeps  moving  the  e-portfolio  service  system  forward  to  meet  new  requirements  of  emerging 
educational  methodologies  and  circumstances. 
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4.2  Pervasiveness 

Fundamentally,  the  restful  e-portfolio  services  need  to  be  accessed  through  basic  HTTP  methods. 
Consequently,  the  widely  support  of  HTTP  client  makes  it  feasible  to  perform  e-portfolio  functions  diversely 
and  pervasively.  This  is  a critical  feature  making  the  e-portfolio  service  system  be  adaptive  to  the 
ever-changing  learning  approaches  and  contexts. 

Besides  the  pervasiveness,  the  simple  application  interface  utilizing  URI  and  HTTP  methods  also  reduces 
work  load  of  service  provider  significantly  because  of  the  lower  time  complexity  of  algorithms  that  are 
responsible  for  composing  and  decomposing  large  volume  of  messages  between  service  clients  and  providers. 
Obviously,  the  simplicity  of  the  restful  e-portfolio  service  benefit  users  of  mobile  devices  in  particular  due  to 
the  inferior  computing  power  and  communication  bandwidth  embedded  within  constrained  space. 

4.3  Accessibility 

Many  kinds  of  teaching,  learning,  and  administrative  works  need  seamless  collaboration  among  multiple 
participants  using  e-portfolio  services.  For  example,  both  assessment  of  a collaborative  learning  activity  and 
nation-wide  evaluation  or  comparison  of  students'  achievements  need  to  consolidate  contents  of  e-portfolios 
that  are  owned  by  multiple  institutes  and  located  on  different  sites.  Fortunately,  the  common  and 
standardized  interface  of  restful  Web  services  make  it  easier  to  consolidate  resources  from  multiple 
e-portfolio  and/or  other  pedagogical  services.  Actually,  it  is  straightforward  to  access  resources  from  multiple 
service  providers  and  integrate  them  directly  within  any  type  of  HTTP  client,  including  the  application  of 
AJAX  technique  in  most  Web  browsers. 


5.  CONCLUSIONS 

Electronic  learning  portfolio  systems  play  a significant  role  in  tracking  learning  activities  and  assessing 
learners'  achievement,  and  have  been  widely  adopted  by  many  institutes  to  fulfill  other  purposes.  To  meet 
diverse  pedagogic  and  administrative  requirements,  as  well  as  adapt  to  ever-changing  learning  contexts,  ideal 
electronic  learning  portfolio  systems  must  be  agile,  versatile,  and  accessible.  However,  most 
currently-deployed  e-portfolio  systems  that  originally  aimed  to  serve  specific  clients  only,  thus  were  built 
with  monolithic  and  proprietary  approach,  which  make  these  e-portfolio  systems  cannot  meet  the  requirement 
of  being  agile,  versatile,  and  accessible. 

The  present  work  proposed  the  concept  of  learning  portfolio  as  a service,  which  actually  is  an  extension 
of  the  generic  software  as  a service  (SaaS)  concept.  The  most  significant  advantages  brought  to  users  by  SaaS 
include  the  lower  preparation  and  maintenance  cost,  and  the  fairer  payment  (pay-per-use)  scheme.  The  restful 
styled  embodiment  of  a SaaS  is  relatively  favorable  in  terms  of  performance,  comparing  with  its  SOAP-based 
counterpart.  The  present  work  designed  and  implemented  the  proof-of-concept  with  restful  style. 

Looking  into  the  future,  the  design  patterns  and  experiences  learned  in  this  work  could  be  applied  to 
develop  other  online  pedagogical  services.  Although  the  feasibility  of  a restful  learning  portfolio  service  has 
been  confirmed,  the  widely  acceptance  of  it  needs  further  works  on  session  management  and  security.  For  the 
sake  of  scalability,  restful  services  supposed  to  be  completely  stateless.  In  other  words,  using  widely  accepted 
server-side  session  mechanism  violates  the  constraints  of  the  restful  design.  However,  it  is  obvious  that 
access  control  and  session  management  mechanisms  are  vital  to  practical  pedagogical  services. 
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ABSTRACT 

This  research  is  a part  of  an  ongoing  project  for  development  of  an  integrated  student  information  system,  aiming  to 
incorporate  self-adaptivity,  personalization  and  social  navigation,  both  in  the  overall  management  of  university  processes, 
and  throughout  the  course  work.  In  this  paper  the  focus  is  on  the  advancement  of  the  existing  course  recommendation 
framework  to  a more  context-aware  level.  The  model  presented  in  this  paper  aims  to  increase  the  personalization  in 
processes  happening  at  the  beginning  of  the  academic  term  and  the  start  of  the  teaching.  The  first  part  of  the  paper 
presents  recent  development  on  the  modernization  of  the  process  of  term  enrolment  and  the  virtual  academic  adviser 
component.  The  evolution  of  the  virtual  adviser  is  discussed  that  allows  the  student  to  change  and  experiment  with 
different  study  paths  and  specializations  without  affecting  the  overall  length  of  the  studies.  In  the  second  part  of  the 
paper,  a new  framework  is  described  that  is  based  on  the  model  of  knowledge  given  in  referent  curriculum  guidelines 
from  professional  organizations.  This  model  is  than  used  to  offer  course  recommendations  that  are  more  inclined  to  the 
context 
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1.  INTRODUCTION 

Throughout  the  years,  higher  education  in  EU  was  extensively  revised  and  new  curricula  and  managed  study 
processes  were  introduced.  The  most  significant  revision  was  the  alignment  towards  the  Bologna  process  and 
ECTS.  In  each  revision,  more  and  more  options  within  the  study  programs  were  given  to  students. 
Nowadays,  mandatory  courses  are  only  50  percent  of  the  programs,  while  the  rest  is  left  on  the  student  to 
choose.  Elective  courses  are  offered  by  other  study  programs  at  the  same  department,  inter-disciplinary 
courses  from  other  departments,  and  corporate  courses  that  can  also  be  recognized  as  credits.  Einally,  student 
mobility  has  evolved  and  through  EU  programs  such  as  Erasmus,  the  student  can  spend  a short  period  at 
another  university  and  enrol  courses  there.  This  creates  administration  considerations  regarding  the  overall 
quality  of  the  study  process,  and  in  addition  to  that  raises  a significant  burden  on  the  student  - which  path  to 
choose? 

We  addressed  some  of  these  issues  with  profiles  - groups  of  related  courses  that  are  not  enforced,  but 
only  used  to  recommend  paths  towards  a certain  specialization.  Unfortunately,  the  students  are  rarely  in 
position  to  strictly  align  to  the  recommendations.  As  unemployment  rises,  they  might  be  forced  to  work 
during  studies  and  support  themselves  and  their  families,  resulting  in  a decrease  in  the  pace  of  studying  - 
lowering  the  number  of  enrolled  credits.  If  this  is  not  possible,  they  are  usually  in  danger  to  fail  a course. 
This,  creates  even  more  problems  - the  student  can  inadvertently  make  wrong  decisions.  Together  with  the 
complexities  in  the  process  due  to  course  inter -dependencies  and  prerequisites,  it  could  later  turn  out  that  the 
student  has  no  other  choice  but  to  prolong  the  total  length  of  the  studies.  In  fact,  this  case  is  not  a rarity,  since 
many  students  take  5 or  6 years  to  finish  a 4-year  long  study  program.  This  problem  can  be  mitigated,  with 
better  communication  with  the  academic  advisers.  But,  as  the  number  of  students  is  constantly  increasing,  the 
quality  of  the  communication  with  the  assigned  academic  advisers  decreases.  This  is  where  the  need  for 
systemic  tools  becomes  obvious.  Tools  that  will  help  the  students  to  assess  their  situation  on  their  own  and 
act  upon  it. 


419 


ISBN:  978-972-8939-88-5  © 2013  lADIS 


2.  VIRTUAL  ADVISER  AND  COURSE  RECOMMENDATIONS 


During  earlier  efforts,  a generic  model  of  the  study  process  was  built  and  it  was  used  as  a base  to  create  a 
new  system  for  management  of  the  study  processes.  This  system,  presented  in  Ajanovski  (2010),  introduced 
the  following  functionalities:  static  structure  of  the  studies  - study  programs,  subjects  and  curricula;  dynamic 
processes  - course  offering  in  each  term,  online  courses  and  timetables;  and  students  status  - admissions, 
enrolments,  mobility  and  recognition.  In  Ajanovski  (2011)  the  implementation  of  a virtual  academic  adviser 
component  was  discussed.  This  component  was  intended  to  help  alleviate  the  problem  of  course  selection  and 
the  influence  of  the  selection  over  the  future  semesters.  The  virtual  adviser  generates  a forecast  map  of 
possible  future  enrolments  of  the  student,  based  on  his  history  and  enables  the  student  to  experiment  with  the 
most  important  new  scenarios:  change  study  program  and  change  order  of  courses  in  order  to  optimize  and 
balance  the  load  during  the  studies,  and  if  desired,  graduate  as  early  as  possible. 

This  development  gave  the  student  better  overview  over  the  context  of  his  situation  and  enabled  wiser 
choices.  This  also  opened-up  possibility  to  include  social  navigation  elements  and  include  automated  course 
recommendations  based  on  the  history  of  enrolments  at  the  institution.  The  current  implementation  of  the 
latest  virtual  academic  adviser,  that  uses  a course  recommender  engine  is  displayed  in  Fig.  1 . The  focus  is  on 
the  upper  part  where  the  map  of  the  possible  future  enrolments  is  shown,  term  by  term.  The  projection 
extends  until  the  semester  in  which  the  student  would  be  able  to  graduate.  Each  future  semester  has  a label 
with  the  form  +n,  which  means  n-th  semester  in  the  future.  This  effort  was  presented  in  Ajanovski  (2013). 


New  Program:  Software  Engineering  Passed  credits:  78/24 

Active  Program:  Information  Systems  Ongoing  credits:  36/240 

Enrolled  terms  in  the  program:  4 Future  credits:  126  mandatory  -4-  O elective  / 240 

I Software  engineering  IH200S-ECTS 

Max  Credits|40  |{  Suggest  plan 


Figure  1 . Current  state  of  the  virtual  adviser. 


The  plan  that  the  adviser  generates  is  based  on  the  study  programs  definition  and  the  chosen  profile  by  the 
student.  First  the  mandatory  courses  are  taken  into  account  and  each  of  them  is  placed  in  the  earliest  possible 
future  term.  After  this  all  electives  from  the  current  chosen  specialization  are  placed.  In  each  step  courses  are 
arranged  according  to  the  maximum  number  of  credits  per  semester,  and  in  which  semester  they  are  on  offer 
and  prerequisites  are  taken  into  account  to  decide  on  priority.  In  the  end,  slots  are  created  for  all  free -choice 
slots  that  will  fill  the  future  plan  up  to  a total  number  of  credits  required  to  finish  the  study  program. 

In  order  to  increase  flexibility  the  recommender  system  Easyrec  was  chosen,  an  open-source  software, 
where  recommender  engines  can  be  developed,  plugged-in  and  experimented  with.  The  system  allows  easy 
integration  with  any  application  due  to  the  REST  API.  The  historical  data  on  course  enrolments  was  imported 
in  the  system  to  prevent  the  cold-start  problem  and  later,  student  course  grades  were  imported  as  course 
ratings.  This  gave  the  possibilities  to: 

• indicate  popular  course  selections  among  similar  students; 

• recommended  courses  based  on  what  similar  students  passed  with  a good  grade; 

• indicate  recommended  programs  to  switch  to. 

Since  the  mandatory  courses  are  already  laid  out  on  the  map  and  only  elective  courses  are  later  filled-in 
by  the  students,  there  is  no  real  danger  of  misguided  automated  recommendations.  The  only  question  is 
satisfaction  of  the  students  with  the  relevance.  All  course  offers  are  listed  and  top  recommended  courses  are 
annotated  with  an  icon,  so  if  the  student  is  not  satisfied  - the  possibility  to  choose  any  other  course  remains. 
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3.  CONTEXT  AWARE  RECOMMENDATIONS  AND  GUIDELINES 

Initial  experiments  confirmed  that  the  recommendations  are  in  line  with  the  expectations,  but  the  approach 
with  a generic  recommender  based  on  history  of  course  enrolments  and  grades  does  not  help  with  offering 
something  new  and  in  the  general  academic  area  (unless  it  was  popular  in  the  past  with  related  students).  So, 
the  software  would  implicitly  incline  the  students  towards  local  popular  courses  from  the  same  department 
and  related  study  programs.  Since  using  recommender  engines  to  issue  course  recommendations  is  not  a new 
topic  an  investigation  of  some  other  efforts  was  done. 

There  are  many  publications  that  show  that  students  will  use  good  recommendations  and  that  they  can 
lead  to  better  success.  Some  of  them  were  influential  for  this  research.  For  example,  in  O’Mahony  and  Smyth 
(2007),  a system  for  offering  course  recommendations  is  discussed  in  an  environment  where  only  20%  of  the 
study  program  is  elective  and  the  number  of  places  per  course  is  limited.  In  this  research,  the  analysis  is  not 
made  on  specific  instances  of  course  enrolments,  but  on  the  pairs  (code,  student)  where  the  code  is  a token 
from  the  course  code,  usually  signifying  the  academic  discipline.  With  this  the  domain  was  reduced  to  about 
80  disciplines  from  about  2000  courses,  in  order  to  have  better  recommendations.  This  gave  the  idea  to  try  to 
use  knowledge  areas  and  topics  instead  of  courses.  The  authors  Itmazi  and  Megfas  (2008)  analyse  and 
discuss  various  systems  for  recommendations  and  filtering  in  a massive  and  open  course  management 
system,  where  students  are  free  to  choose  among  many  courses.  They  propose  an  integrated  framework  that 
filters  the  list  of  courses  according  to  different  criteria  in  several  steps  using  different  types  of  recommender 
and  filtering  systems.  This  gave  the  idea  to  implement  an  external  recommendation  system  such  as  Easyrec, 
which  could  be  plugged-into  and  used  from  several  applications,  and  for  several  different  measures.  Vialardi 
et  al.  (201 1)  define  the  notions  of  a measure  of  student  potential  and  course  complexity  and  later  this  measure 
is  used  to  forecast  the  success  of  the  student  in  a certain  course.  Based  on  this  forecast,  the  course  is 
recommended  to  the  student  or  not.  This  gave  the  idea  to  use  student  grades  as  rating  for  how  relevant  a 
course  is  to  the  success-fullness  of  a student. 

To  disperse  the  interests  to  a wider  area  of  specializations  and  inter-disciplinary  courses,  we  decided  to 
try  an  alternative  approach  with  a more  general  model  of  the  knowledge,  that  can  be  mapped  to  an  existing 
model  of  studies,  so  that  students  could  seek  courses  that  are  from  the  general  academic  area,  or  which  push 
for  similar  learning  goals.  This  is  similar  approach  to  the  one  investigated  by  O'Mahoney  and  Smyth,  the 
difference  is  that  a more  generic  model  is  used  that  is  codified  universally,  so  it  can  be  used  in  an 
inter-university  environment.  The  model  we  start  with,  as  the  initial  building  block,  is  the  concept-model  of 
the  curriculum  architecture  used  in  the  documents  on  ACM  and  AIS  curriculum  guidelines  by  Topi  et  al. 
(2010).  This  model  is  used  to  define  the  structure  of  curricula  proposed  by  ACM  and  AIS  for  undergraduate 
and  graduate  studies  in  information  systems,  but  since  it  is  a generic  model  it  can  be  used  in  any  academic 
discipline.  The  simplified  concept  model  of  the  curriculum  architecture  presented  in  the  referenced  docu- 
ments is  shown  in  Fig.  2. 


COURSE 

TOPIC 

Title 
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Description 
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KNONA/LEDGEUNIT 

Name 
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LEARNING  OBJECTIVE 

Name 
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1 

KNOWLEDGE  AREA 

Name 

Corevs,  elective 

Figure  2.  Simplified  Concept  Model  of  the  Curriculum  Architecture 


This  model  is  interesting  for  our  problem  since  it  defines  the  relationships  between  knowledge  areas, 
knowledge  units  and  topics  per  unit  on  one  hand,  and  the  proposed  courses  together  with  all  learning 
objectives  in  each  course,  on  the  other  hand.  For  each  discipline,  all  knowledge  areas  are  listed,  with 
knowledge  units  and  topics.  Finally,  for  each  course  comprising  the  guidelines  a mapping  is  given  to  covered 
knowledge  topics.  Since  many  universities  offer  similar  or  sometimes  even  the  same  courses  as  in  the 
guidelines,  it  is  possible  to  define  a map  from  actual  university  courses  on  offer  to  the  codified  knowledge 
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areas,  units  and  topics  from  the  guidelines.  Depending  on  the  detail  that  the  university  has  for  each  course 
and  their  own  study  programs,  this  map  can  be  used  to  build  relations  between  students  and  codified 
knowledge  areas,  and  even  a map  of  interests  per  unit.  Relations  are  formed  for  the  pairs:  (student,  area)  and 
if  given  enough  detail  from  the  student  information  system  database  even  ratings/grades  per  area  could  be 
calculated  with  the  triplets  (student,  area,  average/aggregate  grade). 

This  enables  the  usage  of  a recommender  engine  to  reach  indirectly  to  courses  that  have  covered  a similar 
set  of  topics,  but  from  another  perspective.  Such  courses  would  not  be  directly  found  as  relevant  with  the 
usual  approach  based  on  history  of  course  enrolments.  The  difference  between  our  original  and  this  approach, 
is  that  the  first  one  gives  recommendations  blindly  (course  content  is  disregarded  and  only  historical  data  is 
taken  into  account),  while  the  second  approach  is  more  tailored  towards  the  generic  pieces  of  content  that 
truly  build  a course  (as  defined  by  the  curricula  guidelines).  In  addition  to  that,  not  only  are  similarities  found 
indirectly  from  the  content,  this  is  done  in  a structured  way  - via  comparison  to  a reference  structured  model 
of  curricula,  which  gives  the  possibility  to  perform  structured  navigation  through  the  recommendations  and 
enable  the  student  to  choose  an  elective  course  with  broader  sense  of  its  context. 


4.  CONCLUSION  AND  FUTURE  WORK 

The  observations  during  the  experimental  tests  with  a generic  recommender  engine  conclude  that  it  is  mainly 
useful  for  offering  courses  that  are  within  the  same  department,  or  courses  that  are  unrelated  but  are  strongly 
popular.  The  system  could  skip  entire  sets  of  courses  that  have  similarities,  but  were  enrolled  by  entirely 
different  populations  of  students  from  two  different  departments.  This  proposal  takes  into  account  a more 
structured  approach  directly  related  to  reference  categorization  of  course  content  and  uses  it  to  analyse 
historical  data  and  offer  recommendations  within  context.  Since  the  system  is  only  used  in  positions  where 
the  student  has  a free  choice  and  in  such  situation  bad  recommendations  could  not  hinder  the  pace  of 
studying  of  the  student,  there  are  no  negative  side-effects.  The  measure  of  success  of  the  recommendation 
system  has  yet  to  be  discussed,  as  at  the  time  being  it  can  only  be  evaluated  by  the  subjective  satisfaction  of 
the  students  from  the  issued  recommendations.  The  success  of  the  overall  solution  with  the  virtual  adviser  is 
also  a topic  of  discussion  and  further  research,  since  it  can  be  measured  in  several  ways  - decreasing  total 
number  of  empty  term  slots,  decreasing  average  length  of  prolonged  studying,  etc. 
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ABSTRACT 

This  research  investigates  the  factors  that  affect  a students’  take-up  and  continued  use  of  E-leaming.  A mathematical 
model  was  constructed  by  applying  three  grounded  theories;  Unified  Theory  of  Acceptance  and  Use  of  Technology, 
Keller’s  ARCS  model,  and  Expectancy  Disconfirm  Theory.  The  learning  preference  factor  was  included  in  the  model. 
National  culture  influences  the  learning  preference,  hence  national  culture  will  also  be  considered  in  this  research. 
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1.  INTRODUCTION 

Despite  the  high  level  of  investment  on  E-leaming  system  in  many  universities  around  the  world,  researchers 
have  found  a low  uptake  of  this  technology  amongst  students  and  those  who  do  start  to  use  the  system  opted 
out  later  (Liao  & Lu,  2008).  In  order  to  understand  how  to  increase  the  number  of  new  E-leaming  users  and 
how  to  sustain  them  to  continue  using  the  system,  this  research  has  constracted  a model  for  the  effective 
uptake  and  continued  use  of  E-leaming  in  Higher  Education  Institutions. 


2.  FACTORS  INFLUENCING  THE  UPTAKE  OE  E-LEARNING 

In  E-leaming  uptake  research  area,  most  researchers  ground  their  research  in  the  Technology  Acceptance 
Model  (TAM)  or  the  Unified  Theory  of  Acceptance  and  Use  of  Technology  (UTAUT).  TAM  highlights  the 
importance  of  a user’s  attitude  towards  the  system,  but  does  not  address  adequately  the  social  and  resource 
factors.  Therefore,  we  have  adopted  UTAUT  as  the  grounded  theory  for  this  investigation. 

2.1  Unified  Theory  of  Acceptance  and  Use  of  Technology  (UTAUT)  and  the 
Application  in  this  Investigation 

The  UTAUT  model  asserts  that  an  individual’s  uptake  of  new  technology  is  driven  by  their  intention 
(motivation)  to  take  up  that  technology  (Venkatesh,  Morris,  Davis,  & Davis,  2003).  By  applying  the  UTAUT 
model,  behaviour  intention  factor  will  be  adopted  in  our  proposed  model,  called  as  ‘Motivation  to  use 
E-leaming’  factor,  to  capture  a student’s  motivational  level  for  predicting  E-leaming  uptake.  Additionally, 
the  proposed  model  will  have  the  four  UTAUT  factors  that  influence  student’s  motivation  to  take  up 
E-leaming:  [1]  Performance  expectancy  (PE)  is  defined  as  the  degree  to  which  an  individual  student  believes 
in  the  ability  of  an  E-leaming  system  to  support  them  in  learning  activities  to  achieve  their  intended  learning 
outcome;  [2]  Effort  expectancy  (EE)  is  the  degree  to  which  an  individual  believes  that  the  use  of  that  system 
does  not  require  an  increase  in  effort;  [3]  Social  influence  (SI)  is  the  degree  to  which  an  individual  expects 
that  the  use  of  E-leaming  should  be  encouraged  by  their  important  persons;  [4]  Facilitating  conditions  (FC)  is 
the  degree  to  which  an  individual  student  believes  that  an  IT  resource  exists  to  support  use  of  E-leaming. 
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The  factors  from  UTAUT  model  focus  on  technical  point  of  view.  “E-leaming”  has  two  aspects  to  its 
definition,  Teaming’  (referring  to  learning  environment)  and  ‘E’  (referred  to  technology);  the  former  is 
overlooked  by  researchers  in  the  field,  thus  any  existing  model  could  not  fully  explain  the  uptake  of 
E-leaming  (Chen,  2011).  This  research  adds  “learning  motivation”  to  the  proposed  model.  To  identify 
learning  motivation  factors,  the  ARCS  model  has  been  applied. 

2.2  Keller’s  ARCS  Model  and  the  Application  in  this  Investigation 

The  ARCS  model  asserts  that  four  major  factors  influence  a student’s  learning  motivation,  it  includes 
‘attention’  (A),  ‘relevance’  (R),  ‘confidence’  (C)  and  ‘satisfaction’  (S)  (Keller,  1987).  Two  ARCS  factors 
will  be  not  included  in  the  model  as  E-learning  uptake  factor.  According  to  Keller,  attention  can  be  promoted 
by  arousing  the  learner’s  curiosity  in  what  is  being  taught.  However,  this  research  aims  to  increase  E-leaming 
uptake  in  general;  thus  their  attention  and  curiosity  about  a particular  course  is  not  relevant.  Therefore, 
attention  will  not  be  integrated  in  our  uptake  model.  Similarly,  learning  satisfaction  is  not  included  as  an 
uptake  factor.  Learning  satisfaction  occurs  when  a learner  takes  a part  in  learning  environment  (t^)  and 
achieves  their  desired  outcome  from  a course  (Keller,  1987).  At  the  initial  stage  (tg),  before  a student  takes 
up  E-leaming,  their  satisfaction  with  the  provided  course  in  E-leaming  has  not  manifested.  The  use  of  the 
remaining  two  ARCS  factors  (learning  relevance  and  confidence)  in  this  investigation  is  called  ‘learning 
preference’  factor,  and  be  defined  as  the  degree  to  which  an  individual  believes  that  an  instmctional 
environment  in  E-leaming  (which  includes  content  and  learning  activities)  is  relevant  to  their  goals,  learning 
styles  and  has  confidence  in  their  past  experiences  about  what  is  being  learned. 


3.  FACTORS  INFLUENCING  THE  CONTINUED  USE  OF  E-LEARNING 

A problem  of  high  opt-out  rate  with  E-leaming  was  also  found  in  much  of  the  literature  on  the  subject 
(Lee,  2010).  In  order  to  tackle  this  problem.  Expectancy  Disconfirms  Theory  (EDT)  was  utilised  as  a 
grounded  theory.  EDT  asserts  that  repurchase  motivation  is  primary  influenced  by  a customer’s  satisfaction 
with  prior  use  of  the  product  (Oliver,  1980).  The  principle  seems  to  be  consistent  with  many  researches  in  the 
field,  which  found  that  satisfaction  with  the  E-leaming  system  is  a key  influential  factor  that  leads  the 
E-leamer  to  continue  using  the  system  (Roca,  Chiu  & Martinez,  2006).  So  satisfaction  will  be  used  as  a key 
factor  that  influences  a student’s  motivation  to  continue  using  E-leaming  in  this  investigation.  In  addition, 
Oliver  claims  customer  satisfaction  with  the  product  is  directly  influenced  by  confirmation  of  their  level  of 
expectancy;  discrepancy  between  perceived  product  performance  and  the  initial  expectation.  This  principle  is 
supported  by  many  researchers  in  this  field;  they  assert  that  E-Leamers  will  be  satisfied  with  the  E-leaming 
system  if  the  actual  outcome  is  better  than  their  initial  expectation  (Lee,  2010). 


4.  A PROPOSED  MODEL 

By  applying  the  theories  introduced  earlier,  a model  for  E-leaming  uptake  and  continued  use  is  proposed  (see 
Figure  1).  Before  a student  takes  up  E-leaming  (at  the  time  represented  by  tg),  an  initial  expectation  is 
created  (Oliver,  1980).  From  UTAUT  and  ARCS  model,  a learner  has  five  potential  expectations  (belief) 
toward  E-leaming;  performance  expectancy  (hi),  effort  expectancy  (^2),  social  influence  (b^),  facilitating 
condition  (^4)  and  learning  preference  (b^).  Thus,  the  ‘expectation  of  E-leaming’  constmct  (X  h/)  is  added 
into  the  model  to  aggregate  levels  of  belief  for  each  expectation  (uptake)  factor  and  can  be  expressed  as: 


Expectation  of  E-leaming 
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Figure  1 . A model  of  E-learning  uptake  and  continued  use  in  Higher  Education  Institution 

The  learner’s  motivation  to  take  up  E-learning  is  directly  influenced  by  their  expectation  towards  the 
system  and  the  E-leamer  will  decide  to  take  it  up  if  their  expectation  is  high;  sum  of  beliefs  is  positive 
(+  (Oliver,  1980). 

During  the  consumption  period  (t^),  perceptions  of  the  system  performance  will  be  formed,  measured  by 
the  ‘perceived  performance  of  system’  factor  (the  expression  XPi  )•  Oliver  (1981)  asserted  that  initial 
expectation  is  formed  for  creating  a reference  level  that  the  customer  uses  to  make  a comparison  with 
perceived  product  performance  to  determine  their  level  of  confirmation.  Thus,  in  perceived  performance  of 
system  factor,  there  are  five  factors  based  on  each  expectation,  expressed  as: 
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Then,  the  learner  compares  their  perceived  performance  with  their  initial  expectation  to  determine  their 
level  of  expectancy  confirmation.  Therefore,  the  ‘expectancy  confirmation’  factor  (X  6j  ) was  added  in  the 
model,  and  expressed  as  follows: 
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A learner’s  satisfaction  with  E-leaming  is  a function  of  their  expectancy  confirmation;  learners  are 
satisfied  with  the  system  if  each  actual  performance  is  better  than  each  anticipated  performance.  This  is 
expressed  as: 


Satisfaction  with  E-leaming  = positive  expectancy  confirmation  = + Z (4) 

Fishbein  and  Ajzen  (1975)  assert  that  the  most  immediate  precursor  of  motivation  towards  specific 
behaviour  is  attitude,  which  is  the  function  of  personal  belief  (represented  by  expectation  of  E-learning  factor 
in  our  model)  and  evaluation  of  outcome  (represented  by  expectancy  confirmation  factor  in  our  model). 
Therefore,  the  general  equation  is  expressed: 

Motivation  to  use  = f (Attitude)  = /(Z  (5) 
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By  putting  equations  (4)  into  (5)  and  assuming  that  a student  who  takes  up  E-leaming  will  have  a positive 
belief  toward  E-leaming,  thus  the  motivation  towards  eontinued  use  ean  be  expressed  as 

Intention  to  use  t^  = f (+Attitude)  = /(X(-l-I] !);)(+ Z £;))  = + £2^2  + ^3^3  + (6) 

As  ean  be  seen  from  equation  (6),  satisfaetion  (-f  X 6/)  is  a key  faetor  for  supporting  E-leamer  to  eontinue 
using  the  system;  if  E-leamer  does  not  satisfy  with  the  system  (—  X ) they  will  have  nagative  attitude 
towards  system  whieh  lead  them  to  dropt  out  from  the  system. 


5.  AN  APPLICATION  OF  THE  PROPOSED  MODEL  INTO  HIGHER 
EDUCATION  INSTITUTIONS  IN  EACH  COUNTRY 

The  faetors  diseussed  so  far  have  been  widely  validated  in  both  eastern  and  western  eountries,  and  have  been 
aeeepted  by  many  researehers  as  influeneing  towards  uptake  and  eontinued  use  of  E-leaming;  it  is  likely  that 
these  faetors  would  also  be  applieable  to  solve  problems  of  E-leaming  uptake  and  eontinued  use  in  eaeh 
different  eountry  (Chen,  2011;  Maldonado,  Khan,  Moon,  & Rho,  201 1).  However,  one  faetor  that  may  differ 
between  eaeh  other  eountry  is  learning  preferenee,  whieh  ineludes  learner’s  goal  and  style  (Tetiwat  & Huff, 
2003).  Additionally,  literature  suggests  that  the  main  eause  for  differenee  in  learner’s  learning  goal  and  style 
between  eaeh  eountry  is  eulture  (Oxford  & Anderson,  1995).  Therefore,  in  order  to  tmly  understand  students’ 
learning  goal  and  style  in  eaeh  eountry,  the  National  Culture-Influeneed  Learning  Preferenee  (NCILP) 
dimension  was  eonstmeted  in  this  researeh  by  reviewing  the  literature  (see  Figure  2). 
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Figure  2.  National  culture  influenced  learning  preference  dimension 

The  first  dimension  is  relationship  between  eulture  and  edueational  value.  In  eross-eulture  edueational 
value  field,  the  researehers  agrees  that  eastern  and  western  students  have  different  edueational  values;  the 
former  view  edueation  as  a way  to  elimb  the  soeial  ladder  (soeial  approval)  and  to  get  a higher  salary  job,  the 
latter  is  driven  to  learn  by  their  personal  interest  in  that  subjeet  (Biggs,  1991).  In  order  to  be  approved  by  the 
people  soeiety  (e.g.  parent,  peers)  and  obtain  a good  job.  Eastern  students  are  highly  motivated  to  obtain  high 
grades  and  a university  degree.  Therefore,  students  in  these  eountries  are  exam-direeted  learners.  In  eontrast, 
as  can  be  seen  from  the  figure,  in  Western  countries,  students  are  meaning-directed  learners,  and  study  is 
more  a result  of  personal  interest  (Eaves,  2009). 

The  second  and  third  dimensions  explain  the  relationship  between  culture  and  learning  styles.  It  was 
eonstmeted  by  using  Hofstede’s  four  cultural  dimensions  and  Kolb  Learning  Style.  Upon  the  review  of 
literature  on  differences  between  the  cross-culture  of  learning  styles,  the  results  of  all  research  in  this  area 
confirm  that  if  an  individual  has  a reflective  observation  style,  this  strongly  indicates  that  they  are  from  a 
culture  that  has  high  uncertainty  avoidance,  while  people  who  scored  high  in  active  experimentation  have 
low  uncertainty  avoidance  (Joy  & Kolb,  2009;  Yamazaki,  2005).  In  a culture  that  has  high  uncertainty 
avoidance,  people  fear  failure  and  prefer  tasks  with  a definite  outcome  and  clear  guidelines,  thus  students  in 
this  society  are  more  comfortable  in  a lecture  that  is  a stmctured  learning  situation,  more  than  a scientific 
method  with  trial  and  error  process  (active  experimentation)  (Joy  & Kolb,  2009).  The  ‘concrete  experience’ 
style  of  learning  has  also  been  confirmed  by  Yamazaki  (2005)  and  Joy  & Kolb  (2009)  to  be  higher  in  a 
collectivist  culture.  People  in  collectivist  cultures  are  interdependent,  and  students  will  rely  on  others  to  help 
them  learn  (social  learning)  rather  than  by  a self-contained  ‘abstract  conceptualisation’  style  of  learning. 
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NCILP  dimension  help  universities  in  each  country  to  understand  their  student’s  learning  motivation, 
which  can  be  used  for  constructing  learning  environment  in  E-leaming  to  be  responsive  to  their  student’s 
learning  motivation  need.  However,  only  focusing  on  learning  motivation  is  not  enough.  As  E-leaming  is  a 
piece  of  technology;  students  will  not  take  up  this  technology  if  they  do  not  accept  it.  Thus,  the  technology 
motivation  is  also  needs  to  be  considered  by  the  university.  The  literature  review  found  four  factors;  1)  PE: 
E-leaming  in  the  university  has  to  support  students  to  learn  better  (fast  and  higher  performance)  than  other 
possible  way  of  learning;  2)  EE:  the  system  design  suits  the  student’s  level  of  IT  skills;  3)  SI:  the  student  will 
take  up  the  system  if  their  significant  others  (e.g.  parents,  teachers  and  peers)  encourage  them  to  use;  and  4) 
FC:  necessary  IT  resources  (ex.  computer  and  internet)  need  to  be  provided.  As  can  be  seen  from  the 
proposed  model,  by  achieving  these  mentioned  conditions,  not  only  will  university  students  take  up 
E-leaming,  they  will  continue  to  use  the  system.  This  is  because  the  Expectancy  Disconfirm  Theory  (EDT) 
asserts  that  people  will  continue  using  the  service  if  their  expectations  are  met  (Oliver,  1980). 


6.  CONCLUSION 

The  main  purpose  of  this  research  was  to  constmct  a model  of  effective  uptake  and  continued  use  of 
E-leaming  in  Higher  Education.  The  model  from  this  research  will  fill  in  the  gap  left  by  other  researchers  by 
adding  “learning  motivation”  to  the  model.  Moreover,  the  emphasis  on  cross-national  application  of  the 
model;  differences  between  cultural  aspects  will  be  addressed  to  ascertain  their  influence  on  particular 
students’  learning  styles  and  goals.  In  order  to  complete  the  answer  to  these  research  questions,  future  work 
will  focus  on  validation  using  triangulation;  including  investigator  and  methodological  triangulation. 
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ABSTRACT 

This  paper  deals  with  development  of  logical  structures  for  e-leaming  evaluation.  Evaluation  is  a complex  task  into  which 
many  different  groups  of  people  are  involved.  As  a rule  these  groups  have  different  understanding  and  varying 
expectations  on  e-leaming  evaluation.  Using  logical  structures  for  e-learning  evaluation  we  can  join  the  different 
expectations  of  groups  and  can  evaluate  quality  of  e-learning  with  multidimensional  targets. 

Key  targets  form  the  base  for  a logical  evaluation  structure.  The  evaluation  is  successful  or  positive  only  if  all  key  targets 
were  achieved.  If  only  one  key  target  was  not  achieved  the  whole  process  has  missed  its  target. 

As  a rule  key  targets  can  be  achieved  on  different  ways.  This  can  be  mapped  by  corresponding  sub  targets  which  form  the 
inner  logical  stmcture  of  key  targets.  By  this  method  very  flexible  target  stmctures  can  be  modeled.  If  consensus  has  been 
reached  on  the  logical  target  structure  of  an  e-leaming  project  further  evaluation  can  be  performed  based  on  adapted 
checklists  and  the  use  of  measure  theoretical  methods  as  that  has  been  presented  in  previous  papers  of  authors. 
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1.  INTRODUCTION 

Evaluation  is  playing  a very  important  role  during  development,  implementation  and  realization  of  an 
e-learning  framework.  Educational  evaluation  models  and  approaches  were  developed  by  several  authors: 
D.L.Kirkpatrick  and  J.L.Kirkpatrick  (2006),  and  especially  for  e-learning  evaluation  to  Stufflebeam  (1972), 
Khan  (2004),  Ehlers  et  al.  (2006),  Colace  et  al.  (2006),  Lam  and  McN aught  (2008)  and  Ruhe  and  Zumbo 
(2009). 

However,  the  groups  which  are  involved  into  an  e-learning  framework:  pedagogues,  administrators, 
educators,  multimedia  designers,  managers,  stakeholders,  decision  makers,  tutors,  and  learners  have 
distinguished  ideas  and  expectations  on  evaluation. 

Moreover,  e-learning  is  as  a rule  a work,  time  and  cost  intensive  project  which  needs  development  and 
qualification  from  one  implementation  to  the  next.  It  is  a team  work  result  of  different  groups,  experts  or 
partners.  If  we  try  to  measure  the  quality  of  e-leaming  we  have  to  take  into  consideration  the  interests  and 
expectations  of  all  involved  groups.  Besides  the  groups  mentioned  above  this  can  be  moreover  educational 
institutions  and  financial  backers,  for  instance. 

Eigure  1 emphasizes  that  the  interested  groups  in  e-learning  evaluation  can  have  different  expectations 
and  distinguished  targets.  But  all  these  groups  have  one  common  main  target.  They  want  to  measure  anyhow 
the  quality  of  e-learning  or  want  to  know  how  successfully  an  e-learning  course  is  running. 
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Figure  1 . E-learning  and  evaluation 

To  create  and  develop  e-learning  courses  different  environments  have  to  work  together  (Khan,  2004): 

• Pedagogical  environment 

• Interface  Design  environment 

• Management  environment 

• Ethical  environment 

• Resource  support  environment 

• Institutional  environment 

• Technological  environment 

By  means  of  logical  structures  we  can  evaluate  the  quality  of  an  e-leaming  course  not  only  in  sense  of 
expectations  of  involved  different  groups  we  can  measure  the  quality  of  an  e-learning  course  for  each 
environment,  too. 


2.  LOGICAL  STRUCTURES  OF  KEY  AND  SUB  TARGETS 

By  our  model  it  becomes  possible  to  evaluate  for  all  involved  groups  how  the  corresponding  group  has 
achieved  its  target.  Based  on  an  adapted  logical  structure  we  can  see  how  successful  the  components  of  an  e- 
learning  process  are  running,  where  we  have  to  spend  more  attention  and  what  should  be  improved  for  the 
next  round.  A further  advantage  of  our  model  is  we  can  consider  very  flexible  logical  structures,  if  necessary. 
Simultaneously  it  becomes  possible  to  evaluate  in  a consistent  manner  how  special  single  targets  were 
achieved. 

The  formal  details  of  this  approach  are  presented  by  Uranchimeg,  Hardt  (2011).  Key  targets  are  involved 
into  a logical  structure  via  serial  schemes.  A corresponding  main  target  can  be  achieved  only  if  each  key 
target  of  logical  structure  is  achieved.  If  only  one  of  key  targets  is  missed  the  corresponding  main  target  is 
missed,  too. 

Sub  targets  of  key  target  are  visualized  by  a parallel  schemes  (see  Figure  2).  That  means  if  at  least  one  of 
the  sub  targets  is  achieved  we  will  consider  the  corresponding  key  target  as  achieved.  A key  target  can 
consists  of  several  sub  targets.  But  to  extensive  parallel  structures  should  be  avoided.  In  this  case  we  can  get 
high  evaluation  values  for  a key  target  even  in  cases  when  the  evaluation  values  for  the  sub  targets  are 
uniformly  small.  This  should  be  noted  during  the  design  of  a logical  structure. 

After  definition  of  logical  structure  of  an  evaluation  target  the  evaluation  scores  can  be  computed  by 
means  of  the  calculation  rules  presented  in  Uranchimeg,  Hardt  (2012)  based  on  the  observation  data  obtained 
by  adapted  checklists. 
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Figure  2.  Visualization  of  logical  structure 

For  instance,  the  score  that  an  e-learning  course  whose  logical  target  structure  is  defined  as  it  is 
demonstrated  in  Figure  2 is  calculated  by 


=n(i-n^ 


=1  ;=1 


7=1 


Here  denotes  q*ij  = Q*(Aij^  for  i = 1,  = 1,  the  estimation  value  for  the  score  that  the  single 

target  Aij  has  reached  its  target.  These  estimation  values  are  determined  from  the  checklist  data.  It  holds 
0 < Q*{C)  < 1 as  well  as  0 < qij  < 1.  A value  Q*{C)  = 1 means  the  main  targets  of  a target  structure  C 
have  been  reached  completely,  Q*{C)  = 0 means  the  targets  were  missed.  For  the  details  we  refer  again  to 
Uranchimeg,  Hardt  (201 1). 
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Figure  3.  Logical  structure  of  different  groups 
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The  e-learning  evaluation  process  consists  of  eight  steps  (see  Figure  3):  1)  definition  of  key  targets,  2) 
definition  of  sub  targets,  3)  confirmation  of  evaluation  targets,  4)  creation  of  checklist,  5)  acceptance  of 
checklist,  6)  data  collection  according  checklist,  7)  data  processing,  8)  interpretation  of  evaluation  results  and 
outcomes.  Some  steps  are  subjective  oriented  (characterized  in  Figure  3 by  ‘S’),  some  are  more  objective 
oriented  (characterized  in  Figure  3 by  ‘O’).  Subjective  steps  should  be  repeated  until  agreement  is  reached 
between  stakeholders  or  group  leaders  which  are  involved  into  e-leaming  process. 


3.  CONCLUSION 

The  structure  oriented  scoring  model  is  an  alternative  to  the  frequently  used  linear  evaluation  models.  By 
concentration  to  the  logical  structure  of  target  of  an  e-learning  process  and  calculation  of  the  score  by 
application  of  the  corresponding  rules  of  general  measure  theory  we  don’t  need  the  more  or  less  subjective 
weight  factors  which  are  used  for  linear  models. 

The  presented  model  offers  the  possibility  to  consider  parallel  and  consistent  the  expectations  of  the 
different  groups  which  are  involved  into  development  and  implementation  of  an  e-learning  process.  Hence, 
the  model  can  be  applied  for  global  evaluation  of  an  e-leaming  process  as  well  as  an  evaluation  of  embedded 
processes.  Via  the  consideration  of  group  relevant  logical  sub  structures  it  becomes  possible  to  involve  and  to 
integrate  all  interested  groups  into  an  evaluation  process.  The  corresponding  data  can  obtained  based  one  a 
general  checklist  where  the  data  are  processed  according  to  considered  logical  stractures  of  evaluation 
targets.  This  can  avoid  frequently  observed  conflicts  at  determination  of  priorities  for  evaluation.  Using  one 
checklist  we  can  collect  data  from  learners  about  all  environments. 

The  use  of  model  in  context  of  formative  evaluation  can  help  the  institutions  to  recognize  weaknesses  of 
an  e-learning  process  or  to  monitor  running  processes. 

Moreover,  the  structure  oriented  scoring  model  can  be  used  in  more  general  sense  for  stractural 
evaluation  and  monitoring  of  functionality  of  general  processes  with  a logically  describable  function  target. 
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ABSTRACT 

The  use  of  information  and  communication  technology  has  grown  at  an  unprecedented  rate  and  provides  a revolutionary 
way  of  learning  because  specific  information  is  easy  to  find  on  the  Internet.  One  of  the  technology  approaches  in  learning 
is  e-leaming  or  electronic  learning.  Whereas  in  the  U.S.,  e-leaming  has  become  very  popular  in  the  recent  years  as,  in 
Slovakia  e-learning  is  far  from  being  widely  used  as  most  of  the  universities  in  Slovakia  offer  only  in-class  courses  and 
use  information  technologies  only  to  support  those.  One  of  the  disadvantages  of  elearning  is  that  it  provides  more 
possibilities  for  the  academic  fraud  (Nagi,  2006).  The  purpose  of  this  article  is  to  point  at  issues  of  ethical  misconduct  in 
higher  education  and  to  analyze  the  current  threats  of  ethical  misconduct  of  e-leaming  on  an  example  of  Slovakia.  As  part 
of  our  analysis,  the  answers  to  questions  such  as  whether  the  online  students  committed  more  academic  fraud  compared 
to  the  daily  students  will  be  answered  and  conclusions  will  be  drawn. 

KEYWORDS 

E-learning,  ethics,  academic  fraud,  scholastic  honesty,  ethical  misconduct,  online  environment. 


1.  INTRODUCTION 

The  e-learning  is  learning  eased  by  electronic  technology  and  its  aim  is  to  increase  the  knowledge,  skills  and 
productive  capabilities  of  the  learners  (Ismail,  R.  et  al.,  2011,  p.  49-52).  It  serves  as  a substitute  for  the 
traditional  classroom  setting  and  offers  education  with  fewer  space  or  time  limitations,  education  in  which 
discrimination  against  age  and  race  is  almost  non-existent,  the  record  keeping  much  easier  and  discipline 
problems  kept  to  a minimum  (Njenga,  J.,  Fourie,  L.  2010,  p.  199-212).  Despite  of  its  undisputable  benefits 
many  people  view  e-learning  as  inferior  in  quality  in  comparison  with  the  traditional  classroom  setting.  The 
critics  often  argue  that  the  lack  of  personal  contact  causes  low  motivation  of  students.  They  also  claim  that 
the  lack  of  control  causes  that  students  do  not  do  their  tasks  properly  (Khoury  A.  H.  et  al.,  2011,  p.  53-56). 
Another  disadvantage  is  that  e-learning  provides  more  possibilities  for  the  academic  fraud  as  Nagi  (2006) 
states  that  it  is  easier  to  cheat  online  than  face  to  face. 

2.  ETHICS  IN  E-LEARNING 

Of  the  problems  confronting  contemporary  universities,  academic  dishonesty  may  be  one  of  the  most  serious 
especially  since  the  online  environment  provides  a more  tempting  environment  for  students  to  cheat  (Pavela, 
1993).  The  survey  conducted  within  U.S  indicates  that  as  high  as  59%  of  U.S.  students  involved  in  e-learning 
programs  admit  to  some  sort  of  academic  fraud  either  “very  often  (27%)  or  “often”  (32%).  Most  of  the 
studies  on  e-learning  indicate  “psychological  distance”  as  a main  problem  of  studying  online.  When  using 
information  technology  we  tend  to  break  the  ethical  rules  because  the  act  feels  less  personal  as  we  cannot  see 
or  hear  the  other  person.  Today  we  cannot  rely  on  the  fact  that  traditional  moral  rules  were  learnt  at  home. 
Students  tend  to  find  „easy  ways“  of  getting  their  degree  and  do  not  feel  guilty  breaking  the  ethical  rules 
(Brown,  T.,  2008).  Pass  (1990),  in  a study  for  the  American  Council  on  Education,  described  early  patterns 
of  inappropriate  behavior  in  e-learning.  He  identified  the  following  categories  of  academic  fraud  in  the 
e-learning  environment:  inappropriate  assistance  on  examinations,  misuse  of  sources  on  papers  and  projects, 
writing  assistance  and  other  inappropriate  tutoring,  misrepresentation  in  the  collection  and  reporting  of  data, 
improper  use  of  academic  resources,  disrespecting  the  work  of  others,  lack  of  protection  for  human  subjects 
in  research,  breaches  of  computer  ethics,  lack  of  adherence  to  copyright  and  copy-protection,  providing 
inappropriate  assistance  to  others  and  lack  of  adherence  to  academic  regulations. “ 
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Whereas  in  the  U.S.,  e-learning  has  become  very  popular  in  the  recent  years  as  indicated  (Allen,  Seaman, 
2010),  in  Slovakia  e-learning  is  far  from  being  widely  used  as  most  of  the  universities  offer  only  in-class 
courses  and  use  information  technologies  only  to  support  those.  Our  survey  revealed  that  the  Slovak  teachers 
and  students  do  not  seem  to  have  much  experience  in  e-learning  as  above  13%  of  teachers  and  17%  of 
students  answered  “I  don’t  know”  when  asked  to  rate  the  e-learning  at  their  particular  university.  Above  30% 
of  teachers  and  25%  of  students  evaluated  e-learning  at  their  universities  as  bad  or  very  bad.* 

Similar  to  other  countries,  e-learning  in  Slovakia  is  especially  tailored  for  students  that  work  during  the 
day  and  want  to  enhance  their  qualifications  by  acquiring  their  degree  in  the  comfort  of  their  homes.  The 
online  students  are  mostly  adults  that  are  employed  at  various  positions  (even  the  top  management)  and  may 
have  families  which  in  many  cases,  puts  their  education  on  a second  or  third  place  of  priorities  in  their  lives 
(Dutton  et  al.,  2002).  Due  to  the  lack  of  time,  online  students  may  be  more  prone  to  ethical  misconduct  when 
compared  to  daily  students.  On  the  top  of  that,  the  online  students  may  be  less  scared  of  the  consequences  (if 
caught)  since  a dismissal  from  their  study  or  a forced  break  in  their  registration  may  not  affect  their  lives  as 
traditional  students’  lives  would  (Krocity,  2013).  Based  on  the  statistics  provided  by  Slovak  university  that  is 
a leader  in  fighting  the  plagiarism  in  Slovakia  (School  of  Management),  we  tried  to  find  out  whether  online 
students  committed  more  academic  fraud  compared  to  the  daily  students  (expressed  as  a percentage  on  total 
number  of  students).  The  following  table  presents  the  violations  of  scholastic  honesty  by  daily  and  online 
students  for  the  period  of  2002-2011. 


Table  1.  Violations  of  scholastic  honesty  by  daily  and  online  students  (2002-2011) 


Academic  year 

Number  of  students 
violating  scholastic 
honesty  rules 

Total  number  of 
enrolled  students 

Percentage  of  students  violating 
the  scholastic  honesty  rules  on 
total  number  of  enrolled  students 

daily 

online 

daily 

online 

daily 

online 

2002/2003 

65 

2 

548 

52 

11,86 

3,85 

2003/2004 

120 

10 

671 

141 

17,88 

7,09 

2004/2005 

59 

13 

640 

264 

9,22 

4,92 

2005/2006 

68 

37 

713 

411 

9,54 

9,00 

2006/2007 

81 

41 

792 

412 

10,23 

9,95 

2007/2008 

78 

46 

836 

465 

9,33 

9,89 

2008/2009 

78 

69 

850 

496 

9,18 

13,91 

2010/2011 

44 

52 

1030 

458 

4,27 

11,35 

2011/2012 

35 

54 

1047 

340 

3,34 

15,88 

Source:  own  survey  based  on  statistics  of  Scholastic  Honesty  Committee,  School  of  Management,  2013 

The  table  above  provides  significant  results  as  the  percentage  of  academic  fraud  cases  in  daily  programs 
has  fallen  down  over  the  years  whereas  it  has  significantly  increased  in  online  programs  in  the  same  time 
period.  We  have  conducted  an  interview  with  the  Head  of  Scholastic  honesty  Committee  Mr.  Krocity  who 
mentioned  that  the  drop  in  the  number  of  cases  of  academic  fraud  in  the  daily  programs  may  have  fallen 
down  due  to  the  more  intensive  means  of  communication  at  school  (especially  teachers,  classmates, 
admission  officers).  On  the  other  hand,  he  admits  that  communication  with  online  students  has  severe  limits 
and  is  - in  many  cases  - limited  to  the  in  person  orientation  at  the  beginning  of  studies  and  several  warning 
emails  and  short  paragraphs  in  the  school’s  academic  catalogue  and  in  the  course  syllabus.  In  his  study,  he 
has  examined  the  sources  of  information  about  the  scholastic  honesty  cases  (students  were  asked  a question 
“how  did  you  find  out  about  the  scholastic  honesty  rules  and  procedures?”)  that  are  presented  in  the 
following  figure. 


' The  questionnaire  was  distributed  by  email  to  more  than  7,818  teachers  employed  at  the  Slovak  universities  and  more  than  9,020 
students  studying  at  the  Slovak  universities  with  the  response  rate  of  20.23%  in  case  of  teacher  respondents  and  15.61%. 
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Source:  Krocity,  2013  (363  respondents) 


Figure  1.  The  source  of  information  on  scholastic  honesty  rules  and  procedures 

Figure  1 reveals  that  the  most  common  sources  of  information  are  teachers,  course  syllabi,  student 
orientation  and  classmates.  Less  common  sources  are  the  VSM  portal  and  school  catalogue.  The  focus  of 
information  dissemination  should  be  aimed  at  teachers  who  are  also  responsible  for  creating  syllabi  for  every 
course. 

As  aresult  of  this  study,  we  believe  that  the  limitations  in  communication  in  online  programs  (especially 
on  the  part  of  teachers)  accompanied  with  the  increased  number  of  online  students  have  caused  the 
percentage  of  students  violating  the  scholastic  honesty  rules  to  grow.  Our  recommendation  is  to  focus  on 
student-teacher  communication  in  the  forums,  especially  using  a multimedia  communication  means  (e.g. 
video,  video-conference,  webinar)  which  would  be  a better  substitute  of  the  traditional  in  class  setting. 

According  to  Krocity  (2003)  the  most  common  way  of  breaching  the  scholastic  honesty  is  the  improper 
citing  or  paraphrasing  which  was  the  main  interest  of  our  analysis.  However,  other  ways  of  conducting  the 
ethical  misconduct  have  been  revealed.  Due  to  the  limitations  of  this  paper,  we  provide  a brief  summary  of 
the  most  common  cases  as  well  as  our  recommendation  to  limit  the  possibilities  of  the  students  committing 
the  fraud  in  table  2. 


Table  2.  Recommendations  to  fight  the  academic  fraud  in  online  programs 


Most  common  ways  of  ethical  misconduct  in  online 
studies  in  Slovakia 

Our  recommendation  to  fight  the  academic  fraud  in  online  programs 

Improper  citing  or  paraphrasing  on  the  research  papers, 
case  studies,  homework  and  on  discussion  boards. 

Focus  on  prevention  by  intensifying  student-teacher  communication  in  the 
forums,  especially  using  multimedia  communication  means.  Establish 
Scholastic  Honesty  Committee  at  your  university.  Arrange  for  a solid  academic 
plagiarism  checker  technology. 

Writing  assistance  of  another  person  on  behalf  of  a 
student  (often  a paid  service  either  through  a personal 
acquaintance  or  even  by  a professional  company 
specializing  in  writing  papers  on  behalf  of  students). 

Support  the  government  action  to  abolish  the  businesses  that  write  papers  on 
the  behalf  of  students  . Examine  the  student  work  carefully  and  pay  attention  to 
the  student's  writing  style.  Check  the  date  of  creation  of  document  as  a listed  by 
MS  Office.  Ask  student  to  present  the  paper  in  a webinar. 

Providing  username  and  password  to  a third  person  who 
contributes  in  the  discussion  forums  on  their  behalf. 

Pose  similar  questions  from  discussion  forums  on  the  test.  Pay  attention  to  the 
student's  writing  style. 

In  case  of  an  online  test,  a student  may  browse  the 
Internet  to  find  the  correct  answers  or  someone  else  may 
take  the  test  on  the  student’s  behalf  logging  under 
student’s  name. 

We  generally  do  not  recommend  the  online  tests  and  suggest  imposing  the 
requirement  of  taking  1-2  on-campus  proctored  exams  or  using  the  professional 
proctor  if  a student  is  at  a distant  location.  However,  if  online  test  is  the  only 
way,  block  the  possibilities  of  opening  another  window  on  student's  computer. 
Have  a large  test  bank  of  questions  so  that  the  students'  questions  are  unique. 
Ask  open  ended  questions.  Ask  student  to  arrange  a 360  web  camera. 

Asking  someone  else  to  sit  for  the  exams. 

Check  the  student's  ID  at  each  exam.  Pose  similar  questions  from  discussion 
boards  on  the  test. 

In  case  the  exam  is  being  proctored  through  a proctor 
abroad,  the  proctor  may  not  be  reliable  (a  personal 
acquaintance  which  is  very  difficult  to  verify). 

Ask  the  proctor  to  provide  the  verifiable  credentials  (e.g.  the  school's  email 
address,  confirmation  from  the  school  manager  on  a letterhead,  business  card, 
etc.).  Verify  the  proctor  by  calling  the  university  or  check  the  name  and  the 
position  on  the  Internet  (e.g.  on  Linkedin,  Facebook,  etc.). 

Source:  own  work;  conclusions  based  on  the  analysis  of  literature  and  own  experience 
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3.  CONCLUSIONS 

In  our  paper,  we  examined  whether  online  students  generally  commit  more  academic  fraud  compared  to  the 
daily  students  in  Slovakia.  We  found  out  that  at  School  of  Management,  the  university  that  has  the  most 
experience  with  fighting  academic  fraud,  percentage  of  academic  fraud  cases  in  daily  programs  has  fallen 
down  over  the  years  (especially  due  to  the  more  intensive  means  of  communication  at  school  in  daily 
programs)  whereas  it  has  significantly  increased  in  online  programs  in  the  same  time  period.  During  the  last 
year,  the  percentage  of  students  violating  the  scholastic  honesty  rules  in  daily  programs  on  total  number  of 
enrolled  students  was  approximately  five  times  higher  than  in  daily  programs  so  we  can  accept  the  hypothesis 
that  the  online  students  committed  more  academic  fraud  compared  to  the  daily  students.  The  most  common 
way  of  breaching  the  scholastic  honesty  is  the  improper  citing  or  paraphrasing.  One  of  the  main 
recommendations  to  fight  the  academic  fraud  is  to  improve  the  teacher-student  communication  in  online 
programs.  The  focus  should  concentrate  on  the  methods  of  credible  deterrence,  which  do  not  necessarily  have 
to  focus  on  punitive  actions.  Raising  the  level  of  awareness  of  scholastic  honesty  through  more  effective 
information  and  knowledge  dissemination  will  create  a climate  of  mutual  trust  that  encourages  unrestricted 
exchange  of  ideas  and  thus  motivates  people  to  reach  their  highest  potential  (Krocity,  2013). 
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ABSTRACT 

Korean  government  aims  to  implement  SMART  education  nationwide,  so  it  is  planning  many  ways  to  provide  digital 
learning  contents.  There  are  some  ways  of  distributing  digital  contents,  and  each  way  has  its  own  characteristics.  Edunet 
is  nationwide  system  for  providing  educational  resource.  Cyber  Home  Learning  System  is  a regional  service  which  offers 
supplemental  learning  materials.  E-textbook  provides  electronic  textbooks  which  are  actually  used  at  learner’s  real  class. 
Digital  Textbook  uses  the  latest  smart  device  and  technology.  Each  approach  has  advantages  and  disadvantages  so  it  need 
to  choose  a different  spreading  method  depending  on  the  e-leaming  environment.  And  through  the  modularization  of 
contents,  it  needs  to  recycle  digital  contents. 


KEYWORDS 
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1.  INTRODUCTION 

Traditionally,  the  textbook  has  been  regarded  as  the  most  important  learning  material  for  teachers  and 
students.  Because  textbook  was  able  to  contain  letters  and  pictures,  a major  representation  of  knowledge, 
most  of  information  was  spread  in  the  form  of  a book  for  a long  time.  However,  there  are  many  ways  of 
knowledge  representation  and  tools  for  those  representations  are  rapidly  updated  in  this  digital  era.  In  recent 
years,  computer  technology  has  been  improved  rapidly  and  it  affects  the  learning  environment.  Because  most 
of  the  educational  materials  are  becoming  computerized,  there  is  a growing  need  of  new  distribution  system 
which  takes  full  advantage  of  these  materials  in  the  learning  environment. 

Republic  of  Korea  is  one  of  the  countries  which  are  most  interested  in  the  e-learning  environment.  To 
implement  SMART  education  nationwide,  Korean  government  is  concerned  about  digital  contents 
distribution.  So  it  steadily  tests  several  methods  with  various  policies  and  some  main  distributing  services  are 
running  nowadays.  The  purpose  of  this  study  is  to  find  common  features  of  e-learning  system  for  SMART 
education  by  analyzing  characteristics  of  running  services.  To  achieve  the  purpose,  four  services  of  digital 
contents  distribution  are  compared  and  analyzed. 


2.  BACKGROUND 

2.1  SMART  Education  Development  Strategy 

Korea  is  one  of  the  most  fast-changing  countries,  especially  in  this  technical  area.  This  social  trend  provokes 
rapid  changes  in  the  educational  media,  learning  environments,  and  teaching-learning  methods.  To  respond  to 
these  fast-pace  shifts  and  to  nurture  the  talent  needed  for  the  future  society,  Korean  government  is  attempting 
to  transition  to  a new  educational  system,  named  ‘SMART  Education’.  SMART  education  aims  to  get 
learners  participate  in  their  learning  process,  build  their  own  knowledge,  and  develop  lots  of  competences  for 
the  future  society.  With  the  latest  technology  and  pedagogical  theories,  it  is  the  primary  goal  to  create  an 
ideal  environment  where  learners  can  learn  by  themselves. 
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In  SMART  education  environment,  media  that  learners  actually  interact  affect  learning  process  and  result 
of  them;  in  other  words,  the  way  of  contacting  educational  materials  can  be  very  important  factor,  as  well  as 
learning  contents.  From  this  perspective,  Korean  government  has  been  interested  in  contents  distribution.  The 
following  services  show  the  ways  of  providing  educational  contents  in  the  non-face-to-face  education. 

2.2  Edunet 

Edunet  is  a comprehensive  online  service  system  for  exchanging  educational  materials.  It  was  established  in 
1996,  served  as  a nationwide  repository  of  teaching  and  learning  materials  for  K-12.  Teachers  can  easily 
search  teaching  resources;  students  can  acquire  necessary  learning  materials.  Parents  and  educational 
administrators  can  make  use  of  most  of  data.  It  also  offers  basic  cyber  lectures  for  K-12  students  so  those 
who  want  additional  study  can  find  useful  lessons  in  Edunet.  Most  of  the  lectures  are  interactive  lectures 
produced  by  the  Adobe  Elash  programs  and  video  lectures. 

2.3  Cyber  Home  Learning  System 

Whereas  Edunet  is  a nationwide  service,  cyber  home  learning  system  is  a supplementary  e-learning  system 
for  regional  primary  and  secondary  students.  All  metropolitan  cities  and  provinces  offer  their  own  cyber 
home  learning  system  since  2005.  Each  system  focuses  on  local  public  education  so  it  offers  contents  which 
are  closely  related  to  the  real  school  classes.  Similar  to  Edunet,  most  of  the  lectures  are  flash-based 
interactive  lectures  and  video  lectures.  It  also  has  personal  learning  management  system  (LMS),  learning 
contents  management  system  (LCMS),  and  many  community  services  for  students. 

2.4  Digital  Textbook 

Digital  textbook  is  the  generic  term  that  includes  recent  endeavors  to  apply  a variety  of  materials  in  the  field 
of  education,  especially  in  public  education.  Around  the  world,  many  countries  committed  to  replace  the 
traditional  textbook  to  learner-centered  media,  and  Korea  is  one  of  them.  Digital  textbook  project  started  in 
2007.  The  concept  of  digital  textbook  is  an  educational  material  with  a combination  of  textbooks,  reference 
books,  workbooks,  dictionaries,  and  multimedia  contents  such  as  video  clips,  animations,  and  virtual  reality. 
And  it  offers  personal  learning  management  system  and  variety  of  external  learning  materials.  The  aim  of 
digital  textbook  is  to  integrate  all  possible  educational  services  into  it  and  make  it  portal  of  SMART 
education.  Until  2012,  digital  textbook  contents  were  provided  in  the  form  of  Windows  and  Linux  program 
which  is  optimized  for  the  tablet  PC  with  touch  screen.  Prom  2013,  digital  textbook  contents  are  being 
reorganized  to  be  accessed  by  any  devices  and  any  operating  system. 

2.5  E-textbook 

E-textbook  is  a transitional  media  between  textbook  and  digital  textbook.  Digital  Textbook  demands  devices 
equipped  with  the  latest  technology  and  ideal  network  environment.  Although  IT  infrastructure  of  South 
Korea  is  quite  excellent,  it  is  hard  to  supply  every  equipment  required  for  digital  textbook  environment  to 
every  student.  So  e-textbook  aims  to  run  on  common  PC  and  mobile  environment  and  provide  digitalized 
books  which  contain  same  contents  as  textbook  actually  used  in  school.  While  using  e-textbook,  teachers  and 
students  can  get  the  time  to  adapt  to  the  digitized  textbook  environment.  Textbook  providers  produce 
additional  e-textbook  with  lots  of  multimedia  contents  to  help  students’  learning,  and  they  provide  e-textbook 
since  autumn  of  2012. 
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3.  COMPARISON 
3.1  Contents 

Table  1 shows  the  status  of  contents  providing  of  four  services.  There  are  a few  things  to  think  in  terms  of 
contents. 

Textbook  is  more  relevant  to  students’  school  work.  But  e-textbook  does  not  support  interactive  contents, 
so  it  can  decrease  their  interest  and  motivation.  Contemporary  educational  system  needs  more  participation 
of  learners. 

Learners  can  produce  their  own  contents  only  in  digital  textbook  environment.  In  digital  textbook 
platform,  students  can  type  answers,  draw  pictures,  capture  screen,  and  save  those  products  at  their  own 
cloud  storage. 


Table  1.  Contents  comparison 


Contents  provider 

Interactive  contents 

Relevance  with 
school  work 

Edunet 

Central  government 

Flash-based  contents 

Low 

Cyber  home 
learning  system 

Local  education  office 

Flash-based  contents 

Average 

Digital  textbook 

Central  government 

Multi(using  touchscreen 
and  sensors) 

High 

e-textbook 

Textbook  provider 
(Local  education 
office) 

Not  support 

High 

3.2  Usability 

Table  2 shows  usability  of  four  services. 

All  services  are  run  on  PC  environment,  but  digital  textbook  needs  tablet  PC  with  touchscreen  to  use  all 
the  features.  E-textbook  is  the  only  one  which  can  be  run  on  mobile  devices  (in  app  style).  All  service  need 
multimedia  and  interactive  contents,  so  basic  media  player  and  Flash  player  is  required. 

Lectures  of  Edunet  and  cyber  home  learning  system  always  need  connection  to  network.  Digital  textbook 
is  basically  download-and-run  service,  but  to  take  advantage  of  all  the  features  it  has  to  be  connected  to 
network,  ‘e-textbook’  is  purely  download-and-run  service  so  it  does  not  need  network. 

Cyber  home  learning  system  and  digital  textbook  offer  LMS  and  personalized  environment. 


Table  2.  Usability 


Device 

Network 

Connection 

LMS 

User 

contents 

production 

Edunet 

PC 

Always  connected 

X 

X 

Cyber  home 
learning  system 

PC 

Always  connected 

O 

X 

Digital  textbook 

Tablet  PC 

Frequently  connect 

o 

o 

e-textbook 

PC,  Mobile 

X 

X 

X 

4.  CONCLUSION  AND  DISCUSSION 

In  conclusion,  e-Ieaning  system  of  Korean  public  education  can  be  divided  into  two  parts.  First,  there  is  a 
system  which  provides  various  learning  materials  to  general  students.  This  system  offers  universal  lectures 
which  were  produced  by  the  central  government  and  it  does  not  support  any  personalized  service  and  LMS. 
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Edunet  and  e-textbook  are  included  is  this  system.  The  second  one  is  for  personalized  learning,  offering  LMS 
and  user-created  contents  management.  It  provides  interactive  contents  which  are  based  on  the  real  school 
textbooks  and  curriculums.  This  system  includes  digital  textbook  and  cyber  home  learning  system 

Technology  is  enhancing  more  and  more,  but  not  the  entire  e-learning  environment  is  perfect.  In  Korea,  a 
variety  of  e-learning  environment  has  been  built  so  learners  can  study  in  any  learning  environment.  In  order 
to  prepare  smart  learning  environment,  we  would  like  to  make  a few  suggestions.  First,  it  is  necessary  to 
construct  the  environment  in  which  learners  can  produce  and  manage  learning  outcomes  rather  than  one-side 
contents  providing.  Learners  are  exposed  to  a variety  of  learning  materials,  so  it  needs  to  present  meaningful 
and  relevant  contents  to  increase  the  effectiveness  of  learning.  Next,  it  should  be  designed  to  be  used  in 
general-purpose  devices.  Digital  textbook  has  lots  of  merits  and  functions,  but  it  needs  special  devices  for 
using.  Nearly  all  students  have  at  least  one  smart  devices  and  uses  them  well,  all  educational  materials  are 
furnished  for  all  smart  devices. 
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ABSTRACT 

Alternative  assessment  techniques  are  essential  for  increasing  student  learning  in  blended  and  online  courses.  Rather  than 
simply  answer  multiple-choice  questions,  students  can  choose  activities  in  an  academic  contract.  By  using  a contract, 
students  will  be  active  participants  in  their  own  learning.  Contracts  add  a dimension  of  authenticity  to  blended  and  online 
course  and  students  can  learn,  retain,  and  transfer  more  of  course  content.  Guidelines  and  an  example  of  a contract  are 
given. 
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1.  INTRODUCTION 

Developing  effective  assessment  for  online  courses  can  present  unique  challenges  for  both  graduate  and 
undergraduate  level  courses.  Because  many  graduate  course  assignments  are  project-based  rather  than 
objective-based,  authentic  assessment  is  becoming  a motivating  method  of  evaluation  in  many  situations.  In 
undergraduate  courses  where  there  is  often  a lot  of  information  that  needs  to  be  transmitted,  the  use  of 
authentic  assessment  combined  with  objective-based  testing  can  provide  a more  meaningful  and  satisfying 
experience  for  students  ( Hamilton  & Carlson,  2010). 

There  are  many  names  for  online  assessment  that  use  real  situations  and  involve  students  in  active  ways. 
Authentic,  alternative,  and  performance-based  are  regularly  are  used  interchangeably.  The  term  alternative 
assessment  seems  to  be  most  appropriate  because  it  is  an  alternative  to  the  usual  objective-based  testing  that 
is  so  prevalent  in  online  and  blended  education  today.  This  paper  will  use  the  term,  alternative  assessment. 


2.  ALTERNATIVE  ASSESSMENT 
2.1  What  is  Alternative  Assessment? 

Alternative  assessment  is  a type  of  assessment  where  students  perform  as  task  rather  than  select  answers  from 
a test  (Maclellan,  2004).  Students  are  involved  in  completing  the  assignments  that  relate  directly  to  course 
content.  These  can  take  the  form  of  group  or  individual  projects,  presentations,  studying  the  community, 
interviews  on  specific  topics,  and  a wide  range  or  other  activities. 

The  quality  of  student  work  is  judged  on  pre-set  criteria  given  to  the  students  ahead  of  time.  These  criteria 
can  take  the  form  of  rubrics  or  specific  guidelines  to  follow.  These  detailed  criteria  provide  guidance  for 
students  and  increase  their  confidence  because  they  know  what  is  expected  of  them. 

Research  on  alternative  assessment  has  shown  that  learning  for  students  is  moved  into  the  active  category 
rather  them  than  passively  absorbing  information.  Because  students  are  part  of  their  learning  experience,  they 
tend  to  spend  more  time-on-task  and,  therefore,  learn,  retain,  and  transfer  more  course  content. 

Students  have  a greater  interest  in  the  material  because  they  are  involved  in  creating  their  own  products. 
Rather  than  simply  reading  online  text  as  lectures  and  watching  PowerPoint  slides,  they  are  constructing  their 
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own  knowledge  and  exhibiting  their  learning  in  creative  and  unique  ways.  Because  online  and  blended 
students  are  more  involved,  more  learning  occurs. 

There  is  a great  depth  of  understanding  and  cognitive  processing  when  students  are  creating  artifacts  that 
demonstrate  what  they  have  learned.  With  alternative  assessment  they  are  not  just  reading  material  online, 
but  instead,  creating  and  choosing  from  a variety  of  activities  that  can  be  judged  by  the  professor  as  an 
indication  of  their  mastery  of  the  subject  (Reeves,  2009). 

To  be  effective,  alternative  assessment  needs  to  address  the  major  concepts  and  objectives  of  a 
curriculum.  In  all  subjects  there  is  a great  deal  of  information.  It  is  imperative  that  professors  determine  the 
critical  objectives  and  then  design  alternative  assessment  measures  to  match  those  objectives. 

As  with  all  good  instruction,  alternative  assessment  requires  more  planning  time  at  the  beginning.  It  is 
essential  to  start  with  sound  instructional  objectives  in  a variety  of  cognitive  levels.  From  those  objectives, 
the  activities  and  assessment  can  be  constructed.  Each  activity  must  be  created  so  as  to  allow  students  to 
demonstrate  mastery  of  the  content  and/or  skills  of  the  course. 

2.2  Benefits  of  Using  Alternative  Assessment 

Alternative  assessment  is  a more  valid  indicator  of  students’  knowledge  and  skills.  It  encourages  students  to 
be  active  participants  in  their  own  learning  and  moves  them  from  a passive  to  an  active  mode.  Rather  than 
being  on  the  other  end  of  the  computer  receiving  information,  they  are  interacting  with  course  content  and 
developing  meaningful  modes  of  inquiry. 

When  students  are  actively  involved  with  their  learning,  they  are  more  likely  to  develop  critical  thinking 
and  effective  problem  solving  skills.  They  are  not  passively  waiting  for  the  weekly  online  multiple-choice 
test  but  are  thinking  of  ways  to  complete  the  alternative  assessment  assignments  that  provide  them  with 
practice  and  active  learning  opportunities. 

Another  benefit  of  alternative  assessment  for  blended  and  online  courses  is  that  it  provides  variety  for 
professors  and  students.  Students  can  choose  among  many  interesting,  course-relevant  activities  to  complete. 
This  ability  to  be  able  to  choose  from  among  activities  they  are  interested  in  builds  confidence  and 
satisfaction  with  the  subject  (Keller,  2010). 

This  type  of  assessment  requires  practice  in  self-monitoring  and  self-evaluation.  Students  are  responsible 
for  their  own  learning  and  this  is  a very  different  type  of  learning  for  many  students.  They  often  feel 
empowered  to  be  part  of  their  own  learning  and  being  able  to  make  a selection  from  among  alternative 
assignments. 

The  ultimate  benefit  is  that  students  learn  and  retain  more.  Because  they  are  active  participants  in  their 
own  learning  they  feel  a connection  to  what  they  are  doing.  Taking  multiple-choice  tests  and  getting  back  a 
number  grade  can  be  somewhat  satisfying  but  when  students  know  they  have  accomplished  a project  or 
activity,  they  tend  to  remember  that  much  longer  and  are  able  to  transfer  the  new  learning  to  future  situations. 

2.3  Practical  Application  of  Alternative  Assessment 

One  alternative  assessment  technique  that  works  well  and  in  online  and  blended  university  graduate  and 
undergraduate  classes  is  Academic  Contracts.  Academic  contracts,  an  alternative  assessment  technique,  are 
applicable  for  all  ages  and  grade  levels.  They  allow  variability  in  teaching  and  learning.  They  build  students’ 
confidence  by  providing  them  with  varied  activities  at  which  they  can  excel.  They  also  address  learning 
styles  by  letting  students  chose  areas  they  prefer  and  can  do  while  enjoying  the  learning  experience.  So  they 
give  everyone  a change  from  the  usual  multiple  choice  or  essay/term  paper  assignments. 

For  many  online  and  blended  classes,  testing  is  not  a good  option  because  there  is  no  security.  Who 
knows  who  is  taking  the  test?  Or  if  they  are  looking  things  up  in  the  book?  Academic  contracts  are  a viable 
alternative  to  objective-based  testing.  Some  people  have  asked,  "If  you  use  academic  contracts,  how  do  you 
know  the  student  is  actually  the  one  who  is  doing  the  project?"  With  on  campus  courses  professors  do  not 
know  that,  either.  But  it  is  more  difficult  to  get  someone  to  do  a huge  project  for  you  than  it  is  to  take  a test. 

Students  learn  a lot  more  about  a subject  doing  projects,  papers,  and  production-type  work.  Quizzes  can 
be  used  as  a self-check,  though.  You  can  also  use  quizzes  as  self-checks  that  are  not  graded.  This  can  help 
build  confidence  because  students  find  out  how  they  are  doing  as  they  go  along.  They  can  gauge  their 
progress  and  see  how  much  they  have  learned. 
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A major  objective  with  academic  contracts  is  to  provide  students  with  real-world  applications  of  skills 
and  knowledge  gained  in  a course.  When  they  can  apply  what  they  have  learned  to  a meaningful  assignment 
they  tend  to  feel  much  more  satisfied  with  their  learning  experience.  By  applying  what  is  learned  to  relevant, 
meaningful  activities,  students  do  learn  and  retain  more  and  are  able  to  successfully  transfer  this  to  other 
situations  (Lee,  2006). 

The  main  components  of  a good  academic  contract  are  listed  below. 

1 . Cover  a wide  range  of  areas  and  processes  - music,  reading,  writing,  drama,  interviews,  observations, 
video,  analysis,  etc.  Many  professors  make  the  mistake  of  having,  for  example,  seven  assignments  and  four 
or  five  of  them  are  writing  assignments.  There  needs  to  be  MORE  variety  than  that.  Not  all  students  learn  the 
same  way  or  like  to  exhibit  what  they  have  learned  in  the  same  way.  They  should  be  provided  with  different 
ways  to  show  what  they  have  learned. 

2.  Provide  explicit  criteria  for  each  option.  This  is  essential.  If  specific  criteria  are  not  included,  students 
will  not  know  exactly  what  to  do.  Very  specific  criteria  for  each  of  your  assignment  in  your  contract  must  be 
included. 

3.  Make  sure  you  have  enough  optional  assignments  for  them  to  be  optional.  For  an  A,  if  you  require  the 
required  activities  plus  four  optional  activities  and  only  have  five  or  six  from  which  to  choose,  that  is  not 
much  option.  A good  guide  is  to  have  twice  as  many  activities  to  choose  from  as  you  require.  If  they  have  to 
do  four  assignments/activities  then  there  should  be  at  least  eight  to  choose  from. 

4.  Include  a space  for  other  activities.  Students  may  think  of  some  excellent  projects  they  can  do  and 
sometimes  ones  that  professors  will  never  think  of.  Always  have  Other  as  an  option  but  specify  they  need  to 
get  it  approved  by  the  professor  first. 

5.  Stagger  the  due  dates  throughout  the  semester  by  indicating  due  dates  on  the  course  schedule.  Do  not 
make  the  mistake  of  having  everything  due  at  the  end.  There  will  be  too  much  to  grade  at  the  end  of  the 
semester. 


3.  CONCLUSION 

Academic  contracts  are  an  effective  method  for  applying  alternative  assessments  methods  in  online  and 
blended  courses.  Contracts  allow  students  to  be  actively  involved  in  their  own  learning  and  address  their 
learning  styles.  Although  creating  academic  contracts  does  take  additional  time  and  planning,  the  results  are 
worth  it  in  terms  of  student  learning  and  satisfaction.  The  main  consideration  is  to  be  creative  and  provide 
students  with  a variety  of  learning  activities.  Below  is  an  example  of  a contract  used  in  a graduate 
educational  psychology  course  along  with  criteria  examples. 


EPY  502  - Principles  of  Human  Learning  Contract 
Signed  contract  due  second  week  of  class 


Name Completion  date  See  syllabus  for  dates 

Required  Activities  - 500  points 

• Create  a one-page  vita  with  a photo  and  upload  to  course  site  (10  points) 

• Five  concept  maps  and  associated  anecdotes  (50  points) 

• Webliography  - four  annotated  sites  on  relevant  course  topics  (40  points) 

• Complete  an  ARCS  motivational  chart  for  a unit  of  study  (50  points) 

• Design,  implement,  and  evaluate  a behavior  modification  plan  (100  points) 

• Design,  implement,  and  evaluate  a self-regulation  plan  (100  points) 

• Online  discussion  participation  (150,  10  points  per  week) 

Optional  Activities  - Group  I - 100  points  each 

• Design  a board  game  that  teaches  concepts  and  principles  of  a course  you  teach  (may  work  w/partner) 

• Analyze  articles  using  specified  format  (six  articles) 

• Produce  an  instructional  video  on  a topic  covered  in  the  text  - must  be  approved 

• Create  an  academic  contract  and  criteria  to  use  with  your  students 
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Group  II  - 50  points  each 

• Interview  a K-12  teacher  about  instructional  strategies  used  in  the  classroom 

• Observe  a K-12  teacher  and  evaluate  use  of  behavioral  and  cognitive  approaches  (two  diff  teachers) 

• Create  a photo  illustration  of  a topic  or  concept  covered  in  the  text 

• Design  and  implement  a three-day/three  hour  cooperative  learning  activity 

Indicate  choice  of  grade  and  circle  optional  activities  you  plan  to  complete 

A = required  + 200  optional  points 

B = required  H-  150  optional  points 

C = required  H-  100  optional  points 


Student  Date  Instructor  Date 

CRITERIA  EXAMPLES 


Video 

5-8  minutes 

detailed  storyboard  MUST  be  approved 

title  screen 

credit  screen 

narration 

introduction 

body  (3-5  scenes) 

summary/ending 

content  accurate 

lighting  - not  burned  out,  too  dark 

camera  work  - not  jumpy 

transitions  - smooth 

Interview 

15  -20  questions 

intro  statement  for  interviewee  giving  purpose  for  interview  and  how  much  time  it  will  take 

summary  statement 

most  interesting  finding 

what  is  related/not  related  to  course 

improvements  for  next  interview 

start  time  and  end  time 

personal  evaluation  of  session  (1  low,  5 high)  with  two  paragraph  narrative. 

two-page  session  narrative  describing  what  you  found  out  and  how  teacher  could  improve 

your  notes  taken  during  the  interview 
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ABSTRACT 

This  paper  presents  a framework  for  the  evaluation  of  concept  maps.  We  focus  on  supporting  the  student  in  dealing  with 
ill-structured  and  complex  problems.  We  argue  that  these  problems  require  the  application  of  the  modularity  approach.  In 
view  of  this,  the  paper  describes  ways  of  providing  student  support  for  the  implementation  of  this  approach  within  the 
context  of  a concept  mapping  activity.  As  a basis  we  make  use  findings  from  the  field  of  brain  science  and  mathematical 
concepts  and  tools  from  the  field  of  network  theory,  in  order  to  present  specific  evaluation  methods,  the  further 
development  of  which  we  consider  extremely  promising. 


KEYWORDS 

Concept  map,  assessment,  modules,  graph  theory. 


1.  INTRODUCTION 

In  recent  years  we’ve  increasingly  witnessed  a transition  from  the  storage  model  toward  the  development  of 
high-level  skills  (Perkins,  2000)  in  thinking  and  creativity.  The  ideas  and  the  theoretical  background 
underlying  this  transition  are  not  new.  They  were  first  shaped  at  the  beginning  of  the  previous  century 
(Dewey,  1925)  and  have  been  further  confirmed  and  specialized  by  younger  researchers  in  the  field  of 
learning.  Cognitive  tools,  such  as  concept  maps,  play  an  important  role  in  these  approaches  releasing  the 
student  from  dealing  with  low-level  mental  functions  and  encourage  a more  creative  way  of  thinking  and 
synthesizing  knowledge. 

This  article  aims  to  contribute  to  this  field  by  presenting  a framework  and  proposing  guidelines  for  the 
assessment  of  concept  maps.  Research  from  three  different  fields  is  made  use  of  in  order  to  create  the 
framework:  a)  the  field  of  learning,  b)  the  field  of  brain  science  and  c)  the  field  of  network  theory.  The  article 
attempts  to  answer  the  following  questions: 

Why  and  how  do  students  learn  through  the  construction  of  concept  maps? 

In  what  ways  could  we  support  students  and  what  kind  of  feedback  and  suggestions  could  we  provide 
them  with? 


2.  LEARNING  AS  STRUCTURE  CREATION 

We  can  educate  students  to  represent  and  organize  the  information  they  receive  more  efficiently.  This  goal 
could  be  achieved  through  either  strong  or  weak  methods.  Weak  methods  refer  to  methods  which  are 
independent  of  content.  Particularly  when  solving  problems  that  deal  with  the  capacity  for  innovation  and 
navigation  within  ill-structured  domains,  the  use  of  such  methods  is  essential.  Strong  methods  refer  to 
methods  which  are  dependent  on  the  knowledge  domain.  Specific  formulas,  algorithms,  classifications,  etc., 
help  us  to  very  quickly  reach  the  solution  of  a problem  on  the  condition,  of  course,  that  the  particular 
problem  is  well  structured. 

Experts  differ  from  beginners  in  the  strong  and  weak  methods  they  possess  and  apply.  Experts  organize 
information  more  efficiently  and  as  a result  the  solution  space  becomes  more  coherent.  In  this  context, 
evaluation  methods  that  allow  us  to  observe  the  process  of  the  creation  of  structure  by  the  student  acquire 
particular  importance.  This  means  observing  possible  weak  and  strong  methods  which  the  student  applied  in 
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order  to  solve  the  problem.  In  the  case  of  concept  mapping:  a)  strong  methods  are  reflected  in  the  schema 
(diagram)  produced  by  the  student  and  are  expressed  mainly  on  a semantic  level,  while  b)  weak  methods 
could  be  considered  to  be  also  portrayable  in  the  characteristics  of  the  structure  produced. 

A structure  may  or  may  not  be  modular.  A non-modular  stmcture  of  the  natural  world  cannot  be 
characterized  as  “well-structured”  unless  it  is  very  simple  (trivial)  or  if  not  assessed  in  relation  to  dynamic 
growth  and  enrichment.  However,  learning  is  an  entity  not  subject  to  closure.  Learning  constitutes  the 
creation  of  structures,  the  main  qualities  of  which  are  the  potential  for  expansion  and  enrichment.  Its  main 
feature  is  the  change  produced  when  the  framework  expands. 

Structures  with  such  features  have  documented  in  various  ways  (mental  constructions,  simulations, 
visualizations  of  complex  structures,  etc.)  that  the  modular  way  of  development  is  more  efficient  and 
effective  (Newman,  2010;  Lipson,  Pollack  & Suh,  2002)  for  a variety  of  reasons  such  as  easy  error  recovery, 
easier  expansion,  etc.  The  human  brain  is  modular  both  structurally  as  well  as  functionally  (Bullmore  & 
Spoms,  2009).  This  feature  allows  it  to  evolve  and  to  enrich  its  structure.  There  are  indications  (Bassett, 
Wymbs,  Porter,  Mucha,  Carlson,  & Grafton,  2011;  Fair,  Cohen,  Power,  Dosenbach,  Church,  Miezin  & 
Petersen,  2009)  that  when  the  brain  undergoes  learning  processes,  it  becomes  more  modular. 


3.  SEARCHING  FOR  STRUCTURE  IN  A CONCEPT  MAP 


3.1  Finding  Modules 

For  the  detection  of  modules,  namely  the  ideas  contained  in  a map,  we  could  build  on  the  progress  that  has 
been  made  over  the  last  decade  in  the  area  of  network  theory,  both  on  a theoretical  as  well  as  on  an 
application  level.  The  obvious  premise  for  finding  modules  is  that  the  concepts  that  compose  the  ideas 
contained  in  a map  are  connected  to  each  other  in  a different  way  than  with  the  meanings  that  are  outside  the 
framework  that  the  idea  defines.  To  be  more  specific,  we  can  make  the  self-evident  assumption  that  there 
should  be  a high  density  of  connections  among  concepts  within  the  idea,  in  relation  to  concepts  that  are 
outside  it.  Based  on  this  distinction,  unlike  the  case  of  text,  ways  of  connecting  ideas  could  be  proposed. 

For  example,  the  most  successful  algorithm  - it  has  been  applied  with  great  success  in  social  and 
biological  networks  to  detect  communities  or,  more  generally  speaking,  for  the  detection  of  structures  which 
are  distinguished  on  the  basis  of  the  different  density  of  links  embedded  within  the  structure,  compared  with 
links  commencing  from  the  structure  and  ending  in  nodes  of  the  remaining  network  - is  the  algorithm 
suggested  by  Newman  & Girvan  (2004),  which  is  based  on  the  concept  of  intermediate  minimum  paths 
(shortest-path  betweenness). 

In  summary,  the  steps  of  the  algorithm  are: 

1 . Count  the  shortest  distances  (paths)  between  all  pairs  of  vertices. 

2.  Calculate  the  number  of  shortest  paths  that  pass  through  each  edge. 

3.  Find  the  edge  with  the  greatest  number  of  paths  and  delete  them  from  the  network. 

4.  After  deleting  repeat  step  1. 

3.2  Assessing  the  Clarity  of  Modules 

As  mentioned  above,  experts  differ  from  beginners  in  terms  of  how  they  organize  their  knowledge.  This 
means  that  the  structures  the  expert  develops  ought  to  be  more  distinct,  i.e.  characterized  by  greater  clarity 
and  precision.  Pellegrino,  Chudowsky  & Glaser  (2001,  pp.  77)  present  the  maps  portrayed  in  Figure  1 as  a 
typical  example  of  concept  maps  that  were  created  by  an  expert  and  a novice  respectively.  By  examining  the 
maps  on  the  level  of  semantics,  it  is  obvious  that  the  best  treatment  for  the  subject  of  motion  comes  from  the 
expert.  However,  it  is  worth  examining  further,  the  clarity  with  which  the  expert  forms  structures  compared 
to  the  beginner. 

The  detection  of  modules  in  the  context  of  a map  should  be  accompanied  by  a valuation  method 
measuring  the  clarity  of  these  structures.  For  example,  a widely  acceptable  approach  in  the  area  of  network 
theory  for  quantitatively  determining  the  clarity  of  structures  is  with  the  values  of  the  modularity  index. 
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Expert 


Figure  1.  Novices’  and  experts’  concept  maps 


The  mathematical  expression  of  this  definition  is: 


k-k  j 

2m 


where  m is  the  number  of  edges  of  the  map,  A,y  the  adjacency  matrix  in  the  rows  and  columns  of  which  is  0 
or  1 depending  on  whether  there  is  an  edge  between  vertices  i and  j,  k and  kj  is  the  degree  of  vertices  i and  j 
(the  number  of  edges  that  start  from  i and  j),  Ci  and  Cj  the  modules  to  which  the  vertices  i and  j belong 
respectively  and  5(c„  c,j  Kronecker  Delta  (for  c,  = 9 equals  1,  otherwise  0). 

The  modularity  indicator  is  equivalently  expressed  as: 


where  the  first  term  in  parentheses  expresses  the  total  number  of  edges  between  vertices  of  the  same  module 
(i  module)  and  similarly,  the  second  term  expresses  the  total  number  of  connections  for  the  same  module,  if 
the  links  were  randomly  distributed. 

Let’s  suppose  that  an  algorithm  (like  the  one  in  the  previous  section)  has  discovered  two  modules.  We 
refer  to  the  modules  that  are  separated  by  the  bold  line,  as  indicated  in  the  figure  1 . 
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Applying  the  index  in  order  to  evaluate  the  clarity  of  structure  of  the  concept  map  of  the  novice  in 
comparison  to  the  expert  concept  map,  we  will  take  the  following  results: 

The  concept  map  of  the  novice  has  16  concepts  and  15  edges  (30  endings  of  edges)  and  there  are  two 
modules  having  10  and  6 concepts.  The  first  module  has  9 edges  (18  endings  of  edges)  between  its  vertices, 
the  second  module  has  5 edges  (10  endings  of  edges)  and  there  is  1 edge  between  the  first  and  the  second 
module. 

The  clarity  of  modules  is  expressed  as: 

Q = ^(e. . -al)  = 18/30  - (19/30)^+  10/30  - (11/30)^  = 0,397 

i 

The  clarity  of  modules,  in  the  second  case,  is  expressed  as: 

Q = ^(e. . -al)  = 16/34  - (17/34)^ 16/34  - (17/34)^  = 0,441 

i 

Applying  the  above  method,  we  can  prove  in  quantitative  terms  that  the  expert  has  developed  more 
distinct  structures  in  relation  to  the  novice.  The  knowledge  of  expert  is  well-structured  in  relation  to  the 
knowledge  of  the  novice. 


4.  CONCLUSION 

Recent  discoveries  in  the  field  of  brain  science  and  the  latest  constructions  of  mathematical  concepts  and 
tools  in  the  area  of  network  theory  constitute  fertile  ground  for  the  development  and  evaluation  of  learning 
activities  as  that  of  concept  mapping.  The  classical  methods  of  assessment  can  be  enriched  and  greatly 
upgraded  in  light  of  discoveries  about  the  operations  of  the  human  brain.  Learner  support  in  open  type 
learning  activities  and  the  development  of  weak  methods  to  solve  problems  can  be  done  with  the  further 
refinement  and  development  of  a modular-based  framework. 
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ABSTRACT 

The  Knowledge  Society  has  altered  the  way  humanity  works,  learns  and  amuses  itself;  from  here  the  rise  of  the  so  called 
e-leaming,  an  educational  modality  whose  "innovation"  has  been  questioned  because  of  the  tendency  to  simulate 
traditional  ways  to  educate.  This  paper  explores  the  concept  of  e-leaming  2.0,  the  implications  of  implementing 
e-leaming  2.0  in  higher  education,  in  terms  of  challenges,  opportunities,  and  student  readiness  for  this  educational 
modality. 


KEYWORDS 

Web  2.0,  E-Leaming  2.0,  Connectivism 


1.  INTRODUCTION 

We  live  in  a knowledge -based  society,  a society  that  revolves  around  information  in  all  its  presentations.  Due 
to  major  advances  in  information  technologies,  society  certainly  relies  heavily  on  information  that  flows  in 
bytes.  People  who  live  in  this  society,  and  do  not  belong  to  the  so-called  net  generation,  had  to  add  a lot  of 
words  and  concepts  to  their  vocabulary:  Internet,  Web  2.0,  blogs,  wikis,  RSS  reader,  folksonomies,  tags, 
podcasts,  online  social  networks,  hackers,  spam,  and  so  on.  However,  for  the  net  generation,  students  who 
were  born  in  the  1980s,  this  learning  have  been  an  easy  process,  one  intrinsic  to  their  own  development.  For 
them,  Twitter,  Skype,  Facebook,  and  YouTube  are  just  tools  of  everyday  life.  In  this  context,  the  academic 
world  is  trying  to  adapt  its  teaching  and  learning  methods  to  the  new  needs  of  the  marketplace,  the 
technological  advances,  and  the  way  students  grasp  these  technologies  in  their  process  of  knowledge 
generation. 


2.  BODY  OF  PAPER 
2.1  Literature  Review 

The  term  Web  2.0  is  used  to  describe  applications  that  distinguish  themselves  from  previous  software 
generations,  by  a set  of  principles  that  take  advantage  of  Web-based  networks.  Such  applications  encourage 
participation;  and  they  are  inherently  and  socially  open.  Even  though  Web  2.0  is  not  characterized  by  a 
change  in  technology,  in  recent  years  the  Web  has  changed  from  being  a reading  Web  to  be  a read-write  Web 
(Borau  et  ah,  2008). 

The  collaborative  nature  of  Web  2.0  is  consistent  with  educational  theories  such  as  constructivism  and 
Connectivism.  This  makes  Web  2.0  applications  attractive  to  students  and  instructors.  Wikis,  blogs,  and 
social  bookmarking  are  now  commonly  used  in  learning.  The  popularity  of  Web  2.0  is  growing  and  so  are  the 
number  and  variety  of  applications  (Borau  et  ah,  2008).  For  McLoughlin  and  Lee  (2008),  the  advent  of  Web 
2.0  invites  us  to  reflect  on  how  social  software  tools  can  transform  industrialized  models  of  learning,  to  a 
model  focused  on  students'  individual  achievement  based  on  collaboration,  network  communication,  and 
interaction.  A new  concept  has  been  growing  as  a result  of  a better  understanding  of  the  possibilities  of  Web 
2.0  in  education:  e-learning  2.0. 
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The  e-learning  2.0  concept  aspires  to  become  the  next  generation  of  e-learning  by  incorporating  the 
theory  of  Connectivism  and  social  software  in  education  (Hauttekeete,  Mechant,  Veevaete,  & De  Wever, 
2007;  Reig,  2008;  Siemens,  2004).  Instructors  are  beginning  to  explore  the  potential  of  blogs,  multimedia 
sharing  services,  and  other  social  programs,  which,  although  not  designed  specifically  for  e-learning,  can  be 
used  to  motivate  students  and  create  new  and  exciting  learning  opportunities  (O'Hear,  2006).  By  using  the 
new  Web  services,  e-learning  has  the  potential  to  become  more  personal,  social,  and  flexible.  For  O'Hear,  e- 
learning  2.0  combines  the  use  of  Web  tools  and  services  that  complement  the  creation  of  fitted  learning 
communities,  such  as  blogs,  wikis,  and  other  social  programs. 

Traditional  e-learning  resembles  the  typical  classroom  with  an  instructor  and  students.  The  instructor 
provides  high  quality  content,  accessible  through  a learning  management  system  (LMS).  The  learning 
process  is  carried  out  through  communication  tools  and  interactive  exercises.  The  traditional  way  of  teaching 
is  accomplished  through  new  media  (Ebner,  2007). 

Ebner  (2007)  warns  that  by  simply  adding  Web  2.0  to  e-learning  will  not  take  us  to  e-leaming  2.0.  Eor 
him,  there  is  a third  component  that  we  should  take  into  account:  the  human  factor.  Eirstly,  there  is  a learning 
curve  for  Web  2.0;  these  social  tools  are  relatively  new  and  only  a small  percentage  of  people  know  how  to 
use  them.  Secondly,  changing  the  behavior  of  teaching  and  learning  that  has  been  used  for  decades  will  be 
much  more  problematic  than  generating  new  applications. 

2.2  Challenges,  Benefits  and  Preparation  for  Web  2.0 

With  the  interest  of  understanding  the  implications  of  implementing  e-learning  2.0  at  a college  level,  three 
questions  were  explored: 

• What  are  the  major  challenges  or  problems  that  professors  have  to  face  when  using  Web  2.0  and 
e-learning  tools  for  learning  purposes? 

• What  are  the  major  advantages  that  professors  have  when  using  Web  2.0  and  e-learning  tools  for 
learning  purposes? 

• How  ready  do  you  think  students  are  to  take  advantage  of  the  Web  2.0  and  e-learning  tools  for  learning 
purposes?  In  spite  of  the  common  beliefs  of  young  generations,  do  you  think  students  have  the 
knowledge  for  using  such  tools  in  an  effective  way? 

2.2.1  Challenges  for  the  Implementation  of  e-learning  2.0 

The  major  challenge  in  implementing  e-learning  2.0  is  not  so  much  the  development  of  students'  computer 
science  skills,  but  the  development  of  general  skills. 

To  the  challenge  of  promoting  e-learning  2.0: 

• Develop  an  attitude  of  innovation.  Try  to  really  change  the  current  academic  paradigm,  in  which  the 
professor  normally  is  the  one  producing  knowledge  and  the  students  are  receptive  of  it;  and  not  just  do 
the  same  paradigm  with  modern  tools.  A common  temptation  is  to  implement  what  is  new  out  there  but 
still  in  the  traditional  way,  so  professors  tend  to  do  what  they  always  do  now  through  new  technology. 

• Construct  a good  instructional  design.  The  key  to  efficient  use  of  new  technologies  lies  in  the  capacity  to 
make  base  what  is  taught  on  an  appropriate  instructional  design.  Networks  and  Web  2.0  per  se  are  not 
merely  academic  tools,  with  the  exception  that  a user  directs  them  to  achieve  a learning  objective.  It  is 
even  possible  professors  get  lost  if  they  are  not  capable  to  give  students  the  proper  guidelines  to  where 
we  want  to  take  them,  to  where  we  want  to  lead  them. 

• Help  students  to  focus  on  the  academic  subject  matter.  The  challenge  is  to  make  things  not  related  with 
the  class  appear.  The  key  can  be  found  into  taking  advantage  of  the  playful  environment,  while  assuring 
the  effectiveness  of  learning. 

• Have  the  capacity  of  using  technological  tools  of  Web  2.0.  It  is  important  that  professors  know  how  to 
use  Web  2.0  tools  in  an  academic  environment,  since  in  that  way  students  will  fulfill  their  learning 
objectives. 

In  relation  to  e-learning  challenges  faced  by  academic  administrators: 

• Provide  access  to  Web  2.0  tools.  One  in  which  all  students  have  access  e-leaming  2.0  tools,  through 
sufficient  bandwidth  to  use  multimedia. 

• Have  an  open  mind  regarding  implementation  of  new  technologies.  Sometimes  managers  of  the 
infrastractures  oppose  to  use  that  type  of  technologies  because  they  require  computational  resources. 
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In  relation  to  implementing  e-learning  2.0: 

• Social  skills.  Students  who  engage  in  e-learning  2.0  may  lose  interpersonal  skills.  One  of  the  challenges 
is  that  several  social  skills  do  not  get  lost  but  strengthen.  These  social  skills  that  were  referred  by  the 
participants  included  the  ones  of  communicating,  listening,  perception  and  willingness  to  serve  others. 

• Interest.  The  use  of  Web  2.0  to  arouse  and  maintain  students'  interest  and  engagement  in  class  issues. 

• Intellectual  property  issues.  A challenge  for  professors  is  to  ensure  students  avoid  unauthorized  use  of 
copyrighted  materials  and  infringement  of  others'  privacy. 

2.2.2  Benefits  of  Web  2.0  Tools  in  Learning 

Although  many  challenges  were  identified  in  order  to  implement  Web  2.0  in  learning,  there  are  many 

benefits  that  make  it  worthwhile  to  tackle  such  challenges.  Among  these  advantages  are  the  following: 

• Permanence  of  evidence.  These  tools  allow  students  to  create  a sort  of  learning  evidence  portfolio.  For 
example,  the  use  of  blogs  as  a periodical  portfolio  of  learning  reflections. 

• Communities'  creation.  This  sort  of  tools  is  conducive  to  community  creation,  as  students  go  on 
identifying  for  example,  through  the  profile  or  trough  certain  types  of  content,  similar  interests. 

• Multiple  resources.  Such  tools  provide  access  to  a variety  of  information  resources:  presentations, 
documents,  videos,  images,  etc.  These  resources  often  provide  some  learning  activities. 

• Massive  and  rapid  scope.  Students  often  access  them  several  times  a day,  which  is  not  usually  the  case 
with  traditional  learning  platforms  (LMS)  or  with  the  use  of  email  address  use  for  academic  affairs. 

• Creating  and  sharing  knowledge.  These  tools  help  to  foster  the  desire  to  create  and  share  their  knowledge 
with  peers.  By  using  these  tools  all  peers  see  what  each  student  does  is  important,  because  in  the  process 
the  students  feel  more  involved  in  their  own  learning,  while  contributing  to  the  generation  of  knowledge 
in  the  course  and  sharing  it  with  their  colleagues. 

• Playful  environment.  For  students,  a tool  such  as  YouTube  or  Facebook  makes  them  feel  in  their  own 
environment,  or  at  least  in  a neutral  space,  and  not  in  an  institutional  one,  a place  where  besides 
connecting  with  colleagues  and  friends  for  any  given  social  event,  you  can  also  find  out  about  other 
official  activities. 

2.2.3  Preparing  for  College  Students  for  e-learning  2.0 

In  relation  to  the  question  of  whether  college  students  were  prepared  for  e-learning  2.0: 

• Sometimes,  college  students  are  prepared  for  e-learning  2.0.  College  students  are  ready  for  this  type  of 
learning  because  they  know  how  to  use  technology  and  they  have  the  necessary  attitude  to  take 
advantage  of  it.  The  environment  of  e-learning  2.0  is  natural  to  the  students  because,  in  addition,  the  new 
educative  models  go  on  preparing  them,  for  example,  in  their  own  knowledge  management. 

• Sometimes,  college  students  are  prepared  for  e-leaming  2.0,  but  professors  are  not.  Many  college 
students  are  prepared  for  e-learning  2.0,  but  very  few  professors  are.  The  greatest  challenge  to 
implementation  of  e-learning  2.0  is  more  the  professor  than  the  student.  Besides  the  lack  of  technological 
skills,  many  professors  do  not  have  the  necessary  attitude  needed  for  this  type  of  learning. 

• Sometimes,  college  students  are  not  prepared  for  e-learning  2.0.  Although  college  students  have  no 
problems  in  terms  of  technology  skills,  professors  perceive  them  to  lack  sufficient  maturity  to  cope  with 
the  responsibility  of  e-learning  2.0. 


3.  CONCLUSION 

The  e-leaming  2.0  implies  that  those  who  learn,  rather  than  the  those  who  teach,  are  the  ones  who  contribute 
with  content  and  discuss  those  contributions  with  their  contacts,  whether  these  are  classmates,  professors  or 
friends;  engages  their  contacts  as  repositories  of  information;  and  socializes  to  strengthen  bonds  of  trust  with 
their  contacts.  Above  all,  learning  is  self-managed,  with  students  deciding  what  is  good  for  them  to  learn  and 
identifying  who  to  contact  regarding  that  knowledge. 
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Any  person,  registered  or  not  in  an  educational  institution,  can  take  advantage  of  Web  2.0  technologies, 
and  specifically  in  online  social  networks  to  favor  their  learning.  For  this  matter,  a fundamental  requirement 
is  that  the  student  and  professor  have  the  necessary  computer  science  skills.  Some  professors  tend  to  think 
that  college  students  have  computer  science  skills  because  they  heavily  use  information  technologies. 
However,  while  college  students  have  the  ability  to  quickly  learn  the  technological  skills  required,  they  are 
not  always  aware  of  what  technology  skills  are  needed.  Moreover,  the  computer  skills  needed  to  participate 
effectively  in  the  e-learning  2.0  are  not  just  technological.  Cognitive  and  attitudinal  skills  are  also  required.  It 
is  in  this  area  where  professors  face  a greater  challenge. 

Apparently,  now  only  a few  students  understand  the  philosophy  of  Web  2.0  tools.  However,  once  the 
students  understand  how  these  tools  work,  they  will  have  the  ability  to  make  efficient  use  of  them  in  their 
learning  process.  Professors  themselves  must  understand  the  nature  of  these  tools  to  obtain  tangible  benefits.. 

We  face  significant  challenges  if  we  want  to  implement  Web  2.0  tools  in  formal  learning  initiatives  - 
challenges  for  both  administrators  and  professors  to  enable  students  to  make  efficient  use  of  Web  2.0  tools. 
None  the  less,  it  seems  that  there  are  greater  benefits  that  could  be  achieved  by  implementing  e-learning  2.0 
initiatives. 


REFERENCES 

Allen,  G.  (2008).  Practicing  teachers  and  Web  2.0  technologies:  Possibilities  for  transformative  learning  [online 
document].  In:  Dissertations  & Theses:  Full  Text.  Publication  No  AAT  3327101.  Doctoral  Thesis,  Teachers  College, 
Columbia  University,  2009.  [Accessed:  January  9,  2009]. 

Borau,  K.,  Luo,  H.,  Shen,  L.,  Shen,  R.,  Tan,  X.  y Ullrich,  C.  (2008).  Why  Web  2.0  is  good  for  learning  and  for  research: 
Principles  and  prototypes  [online  article].  In:  Proceedings  of  the  17th  International  World  Wide  Web  Conference 
(WWW2008).  ACM  Digital  Library.  [Accessed:  October,  10  2008]. 

Brown,  T.  (2006).  Beyond  constructivism:  Navigationism  in  the  knowledge  era  [online  article].  On  the  Horizon,  4(3), 
108-120.  Emerald  Group  database.  [Accessed:  January  17,  2009]. 

Creswell,  J.  (1994).  Research  design:  Qualitative  and  quantitative  approaches.  Thousand  Oaks,  CA:  Sage. 

Creswell,  J.  (2008).  Research  design:  Qualitative,  quantitative,  and  mixed  methods  approaches.  Thousand  Oaks,  CA: 
Sage. 

Ebner,  M.  (2007).  E-learning  2.0  = e-leaming  1.0  -l-  Web  2.0?  The  Second  International  Conference  on  Availability, 
Reliability  and  Security,  1235-1239  (ARES'07).  [Accessed:  October  10,  2008].  Available  in: 

http://www. informatik.uni-trier.de/~ley /db/indices/a-tree/e/Ebner:Martin.html 

Hauttekeete,  L.,  Mechant,  P.,  Veevaete,  P.  y De  Wever,  B.  (2007).  Multimedia  workshops.  ISMW  '07.  Ninth  IEEE 
International  Symposium  on  10-12  Dec.  2007,  pp.  511  - 516.  Digital  Object  Identifier 

10.1 109/ISM.Workshops. 2007. 91.  lEEEXplore  database).  [Accessed:  October  10,  2008]. 

Lincoln,  Y.  & Guba,  E.  (1985).  Naturalistic  inquiry.  Thousand  Oaks,  CA:  Sage. 

Mcloughlin,  C.  & Lee,  M.  (2008).  Mapping  the  digital  terrain:  New  media  and  social  software  as  catalysts  for 
pedagogical  change  [Online  Article].  Proceedings  Ascilite  Melbourne  2008.  [Accessed:  January  21,  2009].  Available 
in:  http://www.ascilite.org.au/conferences/melbourne08/procs/mcloughlin.html 

O'Hear,  S.  (2006).  E-learning  2.0:  How  Web  technologies  are  shaping  education. 

[Accessed:  October  10,  2008].  Available  in:  http://www.readwriteWeb.com/archives/e-leaming_20.php 

Reig,  H.  (2008).  E-learning  2.0,  bases,  principios  y tendencias.  [Accessed:  September  30,  2008].  Available  in: 
http://www.educaweb.eom/noticia/2008/03/21/elearning-2-0-bases-principios-tendencias-211238.html 

Siemens,  G.  (2004).  Connectivism:  A learning  theory  for  the  digital  age  online  Document].  [Accessed:  September  18, 
2008].  Availability  in:  http://www.eleamspace.org/articles/connectivism.htm 


454 


lADIS  International  Conference  e-Learning  2013 


STORYTELLING:  DISCOURSE  ANALYSIS  EOR 
UNDERSTANDING  COLLECTIVE  PERCEPTIONS  OE 

MEDICAL  EDUCATION 


Yianna  Vovides  and  Sarah  Inman 

Georgetown  University 


ABSTRACT 

Using  discourse  analysis,  the  goal  of  this  exploratory  project  was  to  determine  what  practitioners  of  medical  education  in 
Sub-Saharan  Africa  considered  key  achievements,  within  the  scope  of  their  Medical  Education  Partnership  Initiative 
(MEPI)  activities,  after  their  initial  two-year  implementation  efforts.  To  do  so,  a series  of  58  video  stories  were  analyzed 
from  a 2012  symposium  that  took  place  in  Addis  Ababa,  Ethiopia.  Analysis  of  the  video  stories  followed  an  ethnographic 
method  of  analysis.  Through  this  analysis  common  themes  emerged  from  the  collective  discourse,  which  unsurprisingly 
were  tied  directly  to  the  MEPI  themes  themselves.  The  use  of  discourse  analysis  helped  confirm  that  the  project  themes 
were  very  much  aligned  with  actual  practice.  It  also  enabled  a level  of  transparency  of  results  across  all  stakeholders. 
This  approach  is  therefore  recommended  for  informing  and  supporting  future  programmatic  decisions. 


KEYWORDS 

Video  research,  ethnography,  discourse,  medical  education 


1.  INTRODUCTION 

Alongside  decreasing  costs  related  to  video  recording  devices  comes  an  increase  in  the  availability  of 
audiovisual  qualitative  research  methods  for  data  collection  and  analyses.  In  the  past  decade  there  have  been 
major  advancements  in  making  qualitative  research  more  sophisticated  than  traditional  manual  ways  of 
coding  data  (Walsh,  2003;  Jones,  2007).  These  advancements  offer  an  opportunity  to  examine  collective 
discourse  in  a more  holistic  way.  For  this  reason,  we  have  chosen  to  analyze  the  58  video  stories  of 
participants  who  were  in  attendance  at  the  2"^  Annual  Medical  Education  Partnership  Initiative  (MEPI) 
Symposium  that  took  place  in  2012  in  Addis  Ababa,  Ethiopia.  In  doing  so,  we  aim  to  explore  the  collective 
perceptions  in  relation  to  the  key  MEPI  themes  of  retention,  capacity  building,  research,  communities  of 
practice,  and  sustainability. 


2.  BODY  OF  PAPER 

MEPI’s  2012  2nd  Annual  Symposium,  Building  Partnership  and  Enhancing  Sustainability,  was  designed  to 
bring  together  a network  of  33  African,  17  U.S.,  two  British  and  one  Canadian  medical  schools  and  teaching 
hospitals  as  well  as  policy-makers  from  13  countries  and  local  and  international  partners  to  enhance  the 
sustainability  of  the  initiative’s  outcomes  by  building  synergies  between  the  medical  education  network  and 
other  local,  national,  regional  and  international  stakeholders.  Additionally,  this  conference  focused  on 
innovations  and  best  practices  with  particular  attention  to  capacity  development,  retention  and  regionally 
relevant  research. 

Because  dialogic  interaction  yields  co-constructed  knowledge,  a social  constructivist  approach  can  help 
explain  how  knowledge  and  perceptions  of  knowledge  are  related  within  a particular  context.  The  study  of 
dialogue,  and  the  ways  in  which  interactions  around  others  as  well  as  one’s  own  thoughts,  is  incredibly 
complex  and  critical  to  understanding  collective  intelligence  and  knowledge  transfer  in  that  it  is  an  “issue  of 
language,  talk  and  communication”  focused  on  the  “interplay  of  action  and  interpretation”  (Weick  et  al., 
2005,  p.  409). 
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To  capture  this  complexity  and  understand  the  themes  highlighted  in  the  symposium,  we  selected  a 
grounded  theory  methodology.  Grounded  theory  is  a qualitative  research  method  first  conceptualized  by 
Glaser  and  Strauss  as  a way  of  studying  data  without  preconceived  assumptions.  Glaser  and  Strauss’  (1967) 
strategies  develop  theory  from  research  data  instead  of  from  existing  theories.  An  ethnomethodological 
approach  is  one  in  which  the  world  is  not  studied  in  abstract  terms  but  rather,  “on  the  level  where  reality  is 
being  performed”  (Cruickshank,  2012,  p.  40).  Accordingly,  we  understand  reality  as  a “product  of  our  own 
mental  constructions”  as  it  is  “constructed  by  people  and  society,  as  is  the  case  in  the  theory  of  practice, 
symbolic  interactionism,  phenomenology,  ethnomethodology  and  hermeneutics”  (Cruickshank,  2012,  p.  40). 
This  approach  allows  the  researcher  the  flexibility  of  engaging  in  simultaneous  collection  and  analyses  of 
data. 

To  gather  information  on  how  participants  define  the  achievements  of  MEPI,  we  relied  on  interpretive 
inquiry  through  interviews  and  storytelling.  As  part  of  the  interpretive  approach,  those  who  chose  to  tell  their 
story  were  simply  asked  to  share  their  achievements  over  the  two-year  span  of  MEPI  efforts  in  their  own 
countries.  This  approach  allowed  the  perspective  of  the  storytellers  to  emerge.  By  coding  and  analyzing  the 
individual’s  response,  we  were  able  to  assess  the  collective  knowledge  environment  within  the  symposium 
and  in  relation  to  the  MEPI  themes.  This  is  critical  to  assessing  future  areas  for  development  and  outcomes  of 
a highly  diverse,  international  deliberation — one  in  which  communication  is  critically  important. 

To  analyze  the  discourse  in  the  58  videos,  we  used  NVivo  10,  a qualitative  analysis  software.  Though 
discourse  analysis  has  its  limitations,  it  offers  a transformational  opportunity  as  a driver  for  further 
exploration  in  systems  of  learning  and  knowledge  transfer.  The  58  video  narratives,  which  were  on  average  5 
minutes  long,  were  viewed  and  listened  to  by  a single  researcher  from  the  research  team  and  coded 
accordingly,  using  the  five  MEPI  themes  (retention,  capacity  building,  research,  communities  of  practice,  and 
sustainability)  along  an  iterative  timeline. 

As  Silver  & Pataschnick  (2011)  illustrate,  the  synchronicity  between  transcripts  and  visualization  of  audio 
waveforms  can  help  mitigate  the  limits  of  using  a single  source  of  data,  figure  1 illustrates  the  way  in  which 
transcripts  can  be  viewed  alongside  the  coding  process,  keeping  the  analysis  an  active  process,  furthermore, 
the  transcripts  alongside  coded  material  can  help  further  explain  the  researcher’s  process  for  coding  specific 
information. 
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Figure  1.  Screenshot  of  the  video  analysis  layout  in  NVivo  10 

We  approached  the  research  analysis  by  coding  for  the  five  core  MEPI  themes:  capacity  building, 
communities  of  practice,  research,  retention,  and  sustainability.  Through  the  process  of  coding,  a node  was 
created  when  a specific  mention  of  one  of  the  core  terms.  In  addition,  after  the  initial  mention  of  the  core 
term,  additional  indicators  (sub-nodes)  were  coded  within  NVivo  10  which  provided  a much  more  robust 
definition  of  the  core  themes.  The  following  table  (Table  1)  shows  the  thematic  nodes  and  indicators  (sub- 
nodes). 
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Table  1.  Thematic  Nodes  and  Sub-nodes 


Themes  (nodes) 

Sub-nodes 

Retention 

• Faculty 

• Graduates 

• Health  Worker 

Capacity  Building 

• Quality 

• Quantity 

Research 

• Curriculum  development 

• Faculty  Development 

• Grants 

• Research  Training 

Communities  of  Practice 

• Community  Engagement 

• Tacit  Knowledge 

• Curriculum  Development 

• Increasing  Diversity 

• Rural  Training 

• Situated  Learning 

Sustainability 

• Collaboration 

• ICT 

• Partnership 

• Scalability 

• Standards 

2.1  Findings  and  Discussion 

The  analysis  of  the  discourse  resulted  in  a few  additional  key  themes  that  emerged  from  the  data.  These  were 
e-Leaming  and  Medical  Education.  The  five  core  themes  as  defined  by  not  only  the  MEPI  program  but  many 
of  the  principal  investigators  were  ‘communities  of  practice’,  ‘retention’,  ‘capacity  building’,  ‘research’,  and 
‘sustainability.’  Working  with  these  stories  led  to  greater  depth  of  understanding  in  how  these  core  themes 
are  illustrated  in  practice  (see  Table  1).  Additionally,  a few  other  themes  emerged  from  the  data  such  as  an 
emphasis  on  e-Leaming,  which  was  not  initially  cited  as  a core  theme  but  rather  spans  across  the  themes. 
Medical  Education  was  also  coded  as  a general  overarching  theme.  The  sub-nodes  for  e-Learning  included 
ICTs,  and  Monitoring  and  Evaluation.  The  Medical  Education  sub-nodes  were  Eormal  Education,  and 
Specialist  Training. 

Capacity  building  and  research  were  the  themes  that  had  the  most  specific  references.  A word  frequency 
query  showed  that  the  term  ‘research’  was  mentioned  48  times  in  the  transcribed  data.  However,  the  most 
commonly  coded  themes,  or  nodes,  were  ‘retention’  with  37  references  in  20  sources,  ‘capacity  building’ 
with  38  references  in  23  sources,  and  ‘communities  of  practice’  with  33  references  in  23  sources.  Within 
‘communities  of  practices’  terms  coded  as  indicators  were  ‘community  engagement’,  ‘eurriculum 
development’,  increasing  diversity’,  ‘rural  training’  and  ‘situated  learning.’  Among  these,  ‘rural  training’ 
with  25  references  and  ‘community  engagement’  with  20  references  were  most  common.  Eor  retention, 
separate  indicators  were  mentioned  on  the  retention  of  faculty,  graduates,  and  health  workers.  For  ‘capacity 
building’  there  were  more  references  to  quality  (39)  than  quantity  (25).  This  discrepancy  alerted  us  to  explore 
why  this  might  be;  it  became  clear  that  capacity  building  in  terms  of  quality  was  precipitated  by  an  increase 
in  quantity  of  students,  which  explains  an  increased  occurrence  of  references  to  quality  over  quantity. 

Eor  sustainability — the  most  difficult  term  to  define — we  had  to  explore  outside  sources  as  the  specific 
term  was  not  mentioned  frequently  enough  to  build  a definition.  Etymologically,  the  word  is  derived  from 
Latin  sustinere,  which  means  ‘to  hold  up.’  The  most  common  definitions  found  for  the  word  are 
‘maintaining’  and  ‘enduring’.  Historically,  the  term  has  primarily  been  used  to  refer  to  human-environment 
relations  and  sustainable  economic  development  (Ratner,  2004);  however,  in  the  context  of  this  research,  the 
term  was  used  to  refer  to  making  educational  interventions  sustainable  and  enduring.  We  found  that  the  best 
way  to  code  the  term  was  with  its  more  general  definition  as  ‘something  that  endures.’  When  we  ran  a text 
query  on  ‘endure’  and  words  similar  to  endure,  the  most  eommon  result  was  ‘eapaeity.’  This  suggests  that 
capacity  building  and  sustainability  are  two  themes  that  are  closely  related  and  overlap.  In  determining 
sustainability  indicators  (sub-nodes),  we  primarily  looked  for  mentions  of  scalability  and  implementation  of 
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information  and  communication  technologies.  Given  the  ambiguity  of  the  term,  coding  in  this  area  was  the 
most  organic  and  open-ended.  We  found  that  the  most  commonly  noted  factors  in  sustainability  were 
information  and  communication  technologies  (ICTs)  and  partnerships  both  with  18  references.  This  mostly 
included  telemedicine  and  e-learning  initiatives  so  often  the  codes  were  also  found  under  ICTs  in  the 
‘eleaming’  node.  Other  themes  related  to  sustainability  were  collaboration,  patient  care,  which  coincided 
with  communities  of  practice,  scalability,  and  standards. 

The  research  theme  was  coded  for  ‘research  training’  and  ‘curriculum  development’.  Under  research 
training  we  also  included  training  for  responsible  research  methods.  It  became  clear  that  there  are  overlaps 
between  responsible  research  methodologies  and  the  retention  of  faculty.  A possible  explanation  for  this  is 
that  by  providing  faculty  avenues  for  publication  in  world-class  journals,  faculty  are  incentivized  to  remain  in 
their  respective  fields  and  locations.  This  is  especially  true  considering  an  increased  emphasis  on  rural 
training  and  retention  of  faculty  and  graduates  in  rural  areas.  The  analysis  showed  that  a focus  on  rural 
training  and  communities  of  practice  have  led  to  greater  rates  of  retention  of  faculty  in  rural  areas  where 
medical  support  is  critical. 


3.  CONCLUSION 

In  addition  to  functioning  as  a digital  repository  for  individual  accounts,  the  58  MEPI  videos  illustrate  a 
collective  intelligence  that  is  distinct.  By  coding  and  analyzing  each  story,  we  were  able  to  determine  the 
major  themes  as  well  as  identify  areas  in  need  of  greater  clarity.  This  research  informed  a need  for  better 
methods  of  understanding  individuals  within  diverse,  deliberative  groups,  particularly  in  the  area  of 
international  partnerships  for  medical  education.  This  is  especially  important  when  considering  curriculum 
development,  student  retention,  and  capacity  building. 

As  the  purpose  of  an  international  symposium  is  to  collaborate  with  a diversity  of  participants, 
communication  and  transmission  of  knowledge  is  critical  to  the  success  of  the  event  itself  and  its  aftermath. 
In  the  future,  it  would  be  informative  to  compare  the  number  of  codes  from  video  analyses  with  tags  from  the 
initiative’s  website  (http://mepinetwork.org).  In  this  way,  one  could  observe  the  gaps  between  organizational 
understanding  and  actual  implementation  of  individual  efforts.  In  addition,  this  type  of  research  makes 
collective  intelligence  more  transparent  and  enables  a continuing  dialogue  of  key  issues  that  emerge  from  the 
analysis.  As  a result,  such  analysis  supports  knowledge  transfer  and  has  the  potential  to  support  better  future 
decisions  of  stakeholders. 
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ABSTRACT 

The  new  generation  is  digital  natives  and  they  enjoy  using  the  latest  technology  such  as  online  resources,  mobile  phones, 
and  applications.  Linking  technology  and  English  learning  can  be  an  alternative  to  motivate  the  new  generation  to  learn 
English.  The  participants  in  this  study  included  76  freshmen  in  a university  in  Taiwan.  The  major  data  in  this  study 
included  flashcards,  learning  records  and  self-reflection  on  making  vocabulary  flashcards.  This  study  examined  76 
freshmen’s  perceptions  and  practices  of  making  vocabulary  flashcards  on  Quizlet.  Three  major  findings  were  determined. 
First,  participants'  word  selection  was  based  on  not  knowing  the  meanings  of  the  words,  rather  than  by  their  content  or 
majors.  Second,  Space  Race  was  regarded  as  the  most  helpful  activity  in  terms  of  vocabulary  learning.  Third,  "finding  the 
appropriate  definitions  for  words”  was  the  biggest  challenge  when  making  Quizelt  flashcards.  In  order  to  make  more 
effective  vocabulary  and  content  words  learning  through  online  websites  such  as  Quizlet,  three  suggestions  are  made: 
integration  of  different  elements  of  word  knowledge  with  online  word  activity  designs;  direct  instruction  on  specific 
terms;  and  learners'  awareness  of  word  selection. 


KEYWORDS 

Content  words,  Quizlet,  vocabulary  flashcards,  word  knowledge 


1.  INTRODUCTION 

Freshmen  English  is  one  of  the  requirement  courses  in  universities  in  Taiwan.  College  students  in  Freshmen 
English  classes  major  in  different  subjects,  such  as  education,  educational  psychology,  science,  physical 
education,  etc.  Through  the  required  textbooks,  learners  not  only  learn  English  words,  sentence  structures, 
and  reading  strategies,  but  also  content  knowledge.  These  freshmen  read  English  textbooks  and  encounter 
content  words  in  their  disciplines.  Vocabulary  development  is  strongly  related  to  academic  achievement,  so 
systematic  and  comprehensive  vocabulary  instruction  is  necessary  for  freshmen. 

The  new  generation  is  digital  natives  and  they  enjoy  using  the  latest  technology  such  as  online  resources, 
mobile  phones,  and  applications.  Linking  technology  and  English  learning  can  be  an  alternative  to  motivate 
the  new  generation  to  learn  English.  Scholars  have  investigated  the  integration  of  computer  and  technology 
into  vocabulary  instruction  and  its  effectiveness  on  learners’  vocabulary  learning  (Al-Harf,  2007;  Kaur  & 
Hegelheimer,  2005;  Yip  & Kwan,  2006).  Basoglu  and  Akdemir’s  (2010)  study  concludes  that  using 
vocabulary  learning  programs  on  mobile  phones  is  more  effective  to  improve  students’  vocabulary  learning 
than  using  flashcards  on  paper.  These  sixty  college  students  in  Turkey  responded  that  learning  English 
vocabulary  on  mobile  phones  was  effective  and  entertaining.  The  78  college  students  in  China  in  Zhang, 
Song,  and  Burston’s  (2011)  study  had  easy  access  to  mobile  phones,  so  they  had  repeated  exposures  to  and 
frequent  practice  of  the  vocabulary  items  in  a spaced  manner  on  a daily  basis.  Therefore,  the  short-term 
spaced  vocabulary  learning  via  mobile  phones  became  more  effective  than  massed  vocabulary  learning 
through  the  paper  medium.  Moreover,  the  personalized  mobile  English  vocabulary  learning  system  can 
enhance  fifteen  Taiwanese  college  juniors’  English  vocabulary  abilities  and  promote  their  learning  interests. 
More  significantly,  the  personalized  mobile  English  vocabulary  learning  system  provides  a ubiquitous 
learning  environment  for  English  learning  without  constraints  of  time  or  place  by  mobile  devices  (Chen  & 
Chung,  2008). 


459 


ISBN:  978-972-8939-88-5  © 2013  lADIS 


Flashcards  are  sets  of  cards  that  bear  information.  Language  teachers  often  use  flashcards  when 
introducing  new  vocabulary  words.  Flashcards  usually  have  words  on  one  side  and  pictures  or  definitions  on 
the  other  side.  This  study  aims  to  integrate  online  flashcards  into  vocabulary  learning,  particularly  as  regards 
English  content  words.  The  participants  in  this  study  included  76  freshmen  in  a university  in  Taiwan.  The 
major  data  in  this  study  included  flashcards,  learning  records  and  self-reflection  on  making  vocabulary 
flashcards.  This  study  focuses  on  how  freshmen  can  learn  content  words  in  their  specific  disciplines  and 
majors  through  making  vocabulary  flashcards  on  website  Quizlet  and  review  content  words  through  their 
mobile  phones. 


2.  METHOD 

This  qualitative  case  study  aims  to  discuss  freshmen’s  perceptions  and  practices  of  making  vocabulary 
flashcards  on  Quizlet.  This  study  aims  to  discuss  the  following  issues.  First,  how  did  college  freshmen  choose 
content  words?  Second,  how  did  the  exercises  provided  by  Quizlet  help  them  review  words?  Third,  what 
problems  did  freshmen  encounter  when  making  vocabulary  flashcards  on  Quizlet! 

2.1  Setting  and  Participants 

The  study  was  conducted  for  one  semester  from  September  to  December  2012.  The  participants  included  one 
instructor  and  two  freshmen  English  classes  of  76  students  in  a university  in  Taiwan.  The  class  met  two  hours 
per  week  in  2012  fall  semester.  The  majority  of  the  students  were  Education  majors,  followed  by  Chinese 
majors.  Only  one  student  was  from  the  Department  of  Applied  Science. 

2.2  Data  Collection  and  Analysis 

The  major  data  in  this  study  included  flashcards,  learning  records  and  self-reflection  on  making  vocabulary 
flashcards.  In  the  first  class,  the  instructor  explained  the  steps  of  making  vocabulary  flashcards  and  grading 
rubric  as.  Ereshmen  chose  one  topic  covered  in  the  textbook  and  made  words  related  to  that  topic.  They 
practiced  three  learning  activities  provided  on  Quizlet  (Speller  “listen  and  spell,”  Learn  “read  the  definition 
and  spell  out  the  words,”  and  Test)  and  two  games  (Scatter  “match  words  with  definitions”  and  Space  Race 
“speedy  spelling”).  The  instructor  created  a community  on  the  Quizlet  and  all  the  participants  could  read  the 
flashcards  made  by  other  students.  Participants  were  asked  to  visit  flashcards  created  by  other  students  on  the 
same  topic  and  one  different  topic.  Students  could  review  words  either  on  Quizlet  or  through  their  mobile 
phones. 

Qualitative  data  were  categorized  into  units  of  information.  Once  the  data  were  coded,  tables  were 
constructed  to  summarize  the  data  and  allow  it  to  be  checked  for  patterns  and  themes  based  on  the  research 
questions. 


3.  RESULTS  AND  DISCUSSIONS 

The  data  were  analyzed  and  discussed  in  terms  of  word  selection,  activities  on  Quizlet,  and  challenges  and 
problems  on  making  Quizlet  flashcards,  and  learning  vocabulary  via  mobile  phones  and  websites. 

3.1  Word  Selection 

In  terms  of  word  selection,  the  top  reason  why  students  chose  these  words  was  “I  did  not  know  the  meanings 
of  these  words,  (27)”  followed  by  “These  words  were  useful  (20),”  and  “These  words  were  important  and 
marked  in  red  in  the  textbook  (9).”  Other  reasons  why  students  chose  these  words  included  “These  words 
looked  flin  to  me  (6),”  “These  words  were  difficult  (5),”  “I  randomly  chose  these  words  (3).” 

Learners  should  be  taught  how  to  choose  words  for  making  flashcards.  They  do  not  have  to  memorize  all 
words  that  they  do  not  know  from  reading  an  article.  Ereshmen  in  specific  majors  can  focus  on  Tier  2 words. 
Tier  2 words  are  those  high-utility  terms  that  often  change  meaning  in  different  contexts,  e.g.  globe, 
hemisphere,  legend,  or  region.  So  they  can  be  familiar  with  specialized  vocabulary  in  their  field. 
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3.2  Activities  on  the  Quizlet 

While  37  students  found  Space  Race  helped  them  the  most  in  terms  of  vocabulary  learning,  21  students  chose 
Speller.  Students  found  that  they  could  not  only  memorize  vocabulary  words  but  also  practice  typing  skills 
through  Space  Race.  Moreover,  students  liked  Speller,  because  they  could  improve  their  listening  and 
spelling  skills.  Twelve  students  particularly  liked  Test,  because  they  felt  that  Test  helped  them  practice 
vocabulary  through  different  types  of  exercises.  Only  four  students  liked  Scatter  and  two  students  felt  that 
they  liked  all  the  activities  on  the  Quizlet  website. 

While  37  students  found  Space  Race  helped  them  the  most  in  terms  of  vocabulary  learning,  on  the  other 
hand,  36  students  found  Space  Race  helped  them  the  least  in  terms  of  vocabulary  learning.  These  36  students 
complained  that  they  only  practiced  typing  in  Space  Race  and  the  game  seemed  to  have  no  ending.  They 
disliked  Scatter  because  matching  a word  and  its  definition  was  boring.  Moreover,  students  disliked  Test 
because  they  did  not  like  to  be  tested  as  in  the  traditional  paper  and  pencil  test. 

Teaching  or  learning  words  in  isolation  or  only  word  meanings  is  insufficient  in  building  learners’  word 
knowledge,  and  research  demonstrates  that  building  knowledge  requires  more  than  accumulating  facts  about 
specific  elements  such  as  word  definitions  (Baumann  Kammeenui,  1991;  Benjamin  & Crow,  2010;  Nagy 
(&  Scott,  2000;  Stahl,  2005).  Therefore,  teaching  vocabulary  should  go  beyond  word  meanings,  and  include 
linguistic,  social,  and  cultural  factors  of  academic  language.  Teachers  should  provide  definitions, 
translations,  parts  of  speeches,  sentences,  word  families,  synonyms,  same  word  but  different  meanings,  or 
expression  when  teaching  a word  or  academic  language. 

3.3  Challenges  and  Problems  of  Making  Quizlet  Flashcards 

With  regard  to  the  challenges  and  problems  of  making  Quizlet  flashcards  and  playing  games,  36  students 
responded  “No  problem.”  One  student  replied,  “It  was  as  easy  as  eating  rice  in  Chinese  (this  means  it  was  as 
easy  as  pie  in  English).”  The  top  challenge  that  students  faced  while  making  Quizlet  flashcards  and  playing 
games  was  “I  had  difficulties  in  finding  the  appropriate  definitions  for  words,”  followed  by  “I  was  too  slow 
at  typing  words,”  and  “I  did  not  know  how  to  play  the  games  because  Quizlet  was  designed  in  English.” 

Many  words  have  different  meanings  (e.g.,  sage:  a wise  person;  a herb)  and  serve  different  functions  in 
different  sentences,  texts,  and  even  conversations  (e.g.,  “give”  in  “give  a hug,”  or  “give  a presentation”) 
(Nagy  & Scott,  2000;  Stahl,  2005).  When  creating  word  sets  on  Quizlet,  participants  first  typed  in  words  and 
words’  related  definitions  appeared  on  the  screen.  They  could  then  choose  one  definition  from  among  these 
definitions.  Since  many  words  have  different  meanings,  participants  found  it  difficult  to  choose  the 
appropriate  definitions  for  the  words  as  these  words  were  used  in  the  reading  passage. 

3.4  Learning  Vocabulary  via  Mobile  Pbones  and  Websites 

All  76  college  freshmen  visited  the  Quizlet  website  only  three  times  during  this  study  and  15  of  them  visited 
Quizlet  website  bi-weekly  for  reviewing  words  and  playing  word  games.  Only  eight  students  used  mobile 
phones  to  review  vocabulary  words  and  did  the  activities  provided  by  Quizlet  because  they  thought  Quizlet 
was  convenient  and  fun.  The  major  reason  why  57  students  did  not  review  vocabulary  words  through  Quizlet 
on  their  mobile  phones  was  because  their  mobile  phones  did  not  provide  Internet  access.  Seven  students  did 
not  download  this  Quizlet  application.  Four  other  reasons  why  students  did  not  review  vocabulary  words 
through  Quizlet  on  their  mobile  phones  included  “I  liked  to  write  down  new  words  on  my  notebook,”  “I  was 
too  lazy,”  “The  Internet  service  was  too  slow,”  and  “I  did  not  think  I can  improve  my  vocabulary  by  playing 
games  on  Quizlet.” 

Compared  with  traditional  book-based  vocabulary  learning,  vocabulary  learning  through  mobile  phones  is 
more  effective  (Nation,  2001),  because  it  allows  learners  to  be  exposed  to  vocabulary  items  and  motivates 
them  to  review  vocabulary  (Basoglu  & Akdemir,  2010;  Chen  & Chung,  2008).  Such  effectiveness  may  result 
from  the  affordances  of  the  technology  such  as  “immediacy  in  receiving  the  learning  content,  flexibility  and 
portability  of  learning  in  time  and  place  and  very  low  cost”  (Song,  2008,  p.  95).  However,  without  Internet 
access  and  the  relevant  application  on  mobile  phones  prevented  freshmen  in  this  study  from  learning  and 
reviewing  vocabulary  via  Quizlet. 
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4.  CONCLUSION 

This  study  examined  76  freshmen’s  perceptions  and  practices  of  making  vocabulary  flashcards  on  Quizlet. 
Three  major  findings  were  determined.  First,  participants'  word  selection  was  based  on  not  knowing  the 
meanings  of  the  words,  rather  than  by  their  content  or  majors.  Second,  Space  Race  was  regarded  as  the  most 
helpful  activity  in  terms  of  vocabulary  learning.  Third,  "finding  the  appropriate  definitions  for  words”  was 
the  biggest  challenge  when  making  Quizelt  flashcards.  In  order  to  make  more  effective  vocabulary  and 
content  words  learning  through  online  websites  such  as  Quizlet,  three  suggestions  are  made:  integration  of 
different  elements  of  word  knowledge  with  online  word  activity  designs;  direct  instruction  on  specific  terms; 
and  learners'  awareness  of  word  selection. 

This  case  study  involved  only  76  EFL  learners  and  one  English  teacher.  The  small  number  of  participants 
limits  the  findings,  and  means  that  they  are  not  generalizable  to  larger  English  learner  populations.  Based  on 
the  learners'  learning  records,  flashcards,  and  learners'  answers  on  self -reflection,  however,  the  instructional 
procedure  and  suggestions  for  integration  of  online  tool  on  vocabulary  flashcard  learning  and  teaching  do 
provide  practical  suggestions  for  EEL  classroom  practice. 

This  study  focused  on  making  Quizlet  flashcards  as  tools  to  review  and  practice  vocabulary  learning. 
Different  websites  offer  online  tools  for  learners  and  teachers  to  make  flashcards,  such  as  Ediscio,  Study 
Stack,  iKnow,  Study  Blue,  ProProfs,  and  Word  Dynamo.  Different  online  tools  have  different  functions  and 
features.  A further  study  can  focus  on  comparisons  and  contrasts  of  two  vocabulary  apps  on  digital  natives’ 
vocabulary  learning.  Such  study  can  discuss  the  following  issues:  first,  what  features  can  these  online  tools  or 
mobile  phone  applications  provide  in  terms  of  vocabulary  learning?  Second,  what  are  the  strengths  and 
weaknesses  of  online  tools  or  mobile  phone  applications  in  terms  of  vocabulary  learning  and  instruction? 
Third,  how  do  these  online  tools  or  mobile  phone  applications  improve  digital  natives’  vocabulary  learning? 
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ABSTRACT 

In  this  paper  we  research  the  perceptions  of  professionals  towards  the  acceptability  of  online  degree  programs  in  their 
work  profession.  Although  online  education  is  prolific,  its  acceptability  has  been  slow  and  has  often  been  attributed  to  be 
a poor  quality  alternative  to  traditional  brick  and  mortar  education.  In  this  paper  we  attempt  to  understand  the  reasons 
behind  the  lack  of  acceptability  and  set  future  directions  for  remedying  some  of  the  adverse  perceptions.  We  develop  and 
research  an  approach  for  measuring  background,  online  technology  familiarity,  characteristics  of  instruction  and  distance 
as  primary  drivers  that  would  affect  online  education  effectiveness  and  its  resulting  perception. 
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1.  INTRODUCTION 

Online  accessibility  to  higher  education  has  allowed  working  professionals,  military  personnel  in  isolated 
locations  and  those  living  in  rural  areas  to  earn  degrees  that  were  wholly  inaccessible  just  a few  years  ago 
(Grenzky  and  Maitland,  2001;  Li,  2007).  Furthermore  a number  of  traditional  institutions  report  that 
residential  students,  seeking  to  increase  their  course  load,  or  take  advantage  of  the  flexible  scheduling  and 
convenience  of  online  programs,  account  for  a significant  portion  of  their  online  sections  (Camevale  and 
Olsen,  2003;  Rungtusanatham,  et  ak,  2004).  The  objective  of  the  current  study  is  to  assess  the  role  that 
perception  plays  in  the  acceptability  of  online  offerings  in  a higher  education  setting.  Although  technology 
proponents  have  advocated  the  use  of  online  learning  over  the  past  two  decades,  its  acceptance  in  academia 
as  a quality  offering  has  been  very  slow  (Adams  and  Maitland,  2007).  Traditional,  well-regarded  academic 
institutions  have  been  very  skeptical  regarding  its  effectiveness  and  have  converted  few  if  any  degree 
programs  to  online  instruction. 

When  students’  perception  of  quality,  and  their  performance,  were  compared  during  traditional  and  online 
deliveries  in  the  same  course  (the  blended  or  hybrid  model),  it  was  found  that  the  online  segments  were  as 
effective  as  the  traditional  segments  (Kock,  et  ak,  2007;  Meyer,  2003;  Ryan,  2000).  The  next  logical  step  is  a 
comparison  of  online  and  traditional  offerings  of  the  same  course.  A study  by  Gagne  and  Shepherd  (2001) 
compared  two  class  sections  in  an  introductory  graduate  level  accounting  course,  one  section  was  traditional, 
the  other  online.  The  performance  of  both  groups  was  similar.  Furthermore,  student  evaluations  of  the 
respective  instructors  were  similar.  The  only  caveat  was  that  students  in  the  online  section  were  less  satisfied 
with  instructor  availability  than  the  traditional  offering. 


2.  AN  EMPIRICAL  STUDY  OF  PERCEPTION 

The  literature  indicates  that  there  are  several  factors  that  influence  the  effectiveness  of  online  instruction.  One 
of  the  clear  indicators  seem  to  be  exposure  and  experience  in  online  education  (Sonner  1999).  In  order  for 
online  education  to  be  effective  both  instructors  and  students  must  have  experience  with  online  instruction.  It 
is  expected  that  the  perception  towards  online  education  is  influenced  by  experience.  Other  factors 
influencing  the  effectiveness  of  online  instructions  are  technology  background,  familiarity  with  online 
technology,  and  distance. 
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2.1  Online  Education  - A Case  Study 

Around  the  year  1999  the  Commonwealth  of  Virginia  in  the  United  States  asked  one  of  its  large  state 
universities  to  develop  a fully  online  degree  program  in  information  technology  to  meet  the  growing  demand 
for  IT  professionals  in  the  Commonwealth.  The  university  responded  to  the  need  by  developing  a fully 
online  Masters  in  Information  Technology  program.  This  program  is  an  interdisciplinary  degree  program 
consisting  of  courses  in  business,  computer  science,  and  electrical  engineering.  The  curriculum  design  for 
such  a program  required  consideration  to  be  given  to  variation  in  content  and  format  among  the  three 
disciplines.  For  example,  courses  in  computer  science  and  engineering  typically  involve  higher  degree  of 
quantitative  and  scientific  content  than  those  taught  in  business.  Often  computer  science  and  engineering 
courses  rely  heavily  on  lecture-based  presentations  and  hands-on  laboratory  experiments.  Whereas,  business 
courses  involve  greater  level  of  faculty-student  interactions  on  such  things  as  group  presentations  and  case 
analyses. 

Courses  in  the  program  are  designed  to  meet  live  over  the  internet  using  software  called  Centra 
Symposium  allowing  direct  interaction  between  students  and  faculty.  These  sessions  are  recorded  allowing 
students  to  play  and  replay  the  lecture  sessions  as  needed.  Course  materials  are  posted  on  software  like 
Blackboard.  To  create  a classroom  “feel”  the  online  technology  used  for  instruction  enables  student  teams  to 
set  up  discussion  forums.  It  permits  texting  between  the  instructor  and  students  during  class  meetings,  and 
enables  case  and  project  presentations  in  real-time  by  student  groups  using  PowerPoint  slides  or  dynamic 
web  pages.  For  example,  the  Cyber  Security  course  within  the  engineering  discipline  includes  students 
accessing  a virtual  computer  to  hack  a pre-designed  system  (environment).  This  simulation  activity  is 
performed  real-time  by  students  while  simultaneously  participating  in  the  live  class  lectures  delivered  online. 

Initially  there  was  faculty  resistance  to  the  program  based  on  the  perception  that  a fully  online  program 
would  not  be  able  to  meet  desirable  quality  standards.  However,  because  of  the  Commonwealth’s  mandate 
the  program  moved  forward  and  was  successfully  implemented.  The  program  has  enjoyed  several  years  of 
success  and  has  continued  to  grow.  Quality  issues  have  been  resolved  using  technology  and  appropriate 
training  of  faculty.  To  date  over  600  students  have  graduated  from  the  program  and  enrollments  keep 
increasing. 

Based  on  the  success  of  the  online  implementation  of  this  program,  it  was  then  considered  for  possible 
delivery  to  students  in  Mumbai,  India  in  collaboration  with  a reputed  business  school.  In  spite  of  India’s 
rapid  growth  in  the  information  technology  area,  there  has  been  little  acceptance  of  online  degree  programs 
in  that  country.  As  such,  program  delivery  format  was  modified  from  a purely  online  format  to  a hybrid 
approach.  In  this  delivery  method  at  least  30%  of  the  course  is  offered  online  and  the  remainder  is  delivered 
using  traditional  in-class  instructions  by  U.S.  faculty  traveling  to  Mumbai.  Over  the  last  few  years,  following 
the  initial  offering  of  the  program  in  India,  Indian  students  and  faculty  have  developed  a favorable  perception 
toward  online  courses  and  are  willing  to  accept  that  online  programs  can  be  offered  without  jeopardizing 
quality.  As  an  indicator  of  program  quality,  students  in  India  have  been  placed  in  highly  desirable  jobs  and 
are  recruited  into  corporate  positions  that  offer  competitive  salaries  similar  to  graduates  of  highly  respected 
institutions  in  India.  Another  measure  of  success  for  this  program  is  the  exponential  growth  in  the  number  of 
applications  received  from  prospective  students.  Similar  to  their  counterparts  in  the  U.S.,  students  in  India  are 
able  to  participate  in  class  meetings  with  the  faculty  lecturing  online  from  the  U.S.  home  base.  Course 
materials,  classroom  discussions,  lectures  and  presentations,  and  other  requirements  have  remained  identical 
between  the  two  groups  despite  the  online  format  and  geographical  and  cultural  differences. 

Despite  the  favorable  perception  of  the  program  among  applicants  and  students,  and  the  successful 
placement  of  its  graduates  in  the  IT  industry,  it  is  not  yet  clear  whether  the  perception  and  acceptability  of 
online  education  among  corporate  managers  and  executives  who  hire  graduates  of  such  programs  has 
improved.  To  examine  the  perception  of  professionals  with  respect  to  effectiveness  of  online  degree 
programs  data  was  collected  by  trained  investigators  in  data  collection  using  a questionnaire.  One  hundred 
and  fourteen  professionals  agreed  to  participate  in  the  experiment  and  were  interviewed. 
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This  paper  will  analyze  the  perception  of  online  learning  using  data  collected  from  the  subjects  described 
in  the  case  study.  Using  regression  analysis  the  relationship  between  experience,  technology  background, 
familiarity  with  online  learning,  professional  characteristics  and  distance  are  tested  and  analyzed.  Results  of 
the  study  indicate  that  with  the  exception  of  background  in  information  technology  all  other  factors  play  an 
important  role  in  the  perception  of  learners  toward  online  courses.  These  results  will  be  demonstrated  in  more 
detail  in  the  conference  presentation  and  recommendations  will  be  made  for  possible  extensions  to  the 
methodology  used  in  this  paper. 
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ABSTRACT 

This  study  analyzed  effective  test  formats  that  utilized  tablets  for  tests  in  university  information  basic  subjects  in  blended 
learning  environments.  Specifically,  three  types  of  test  were  created:  (1)  multiple-choice,  (2)  fill-in-the-blank,  and  (3)  a 
mixture  of  multiple-choice  and  fill-in-the-blank.  An  analysis  focusing  on  university  students’  approach-avoidance 
tendency  was  conducted  to  see  how  the  question  formats  affected  the  students’  motivations  toward  the  tests.  The  results 
showed  that  regardless  of  the  test  approach-avoidance  tendency,  the  students  had  a low  perception  of  their  understanding 
of  the  content  of  multiple-choice  tests,  even  though  they  reported  that  they  did  not  feel  burdened  in  answering  the 
questions.  Additionally,  although  the  students  had  a high  perception  of  knowledge  retention  with  fill-in-the-blank 
questions,  their  willingness  to  answer  the  questions  and  the  percentage  of  questions  they  answered  correctly  were  low. 
On  the  other  hand,  students  had  a high  perception  of  continuity  and  a high  percentage  of  correct  answers  on  the  test  with 
a mixture  of  multiple-choice  and  fill-in-the-blank  questions.  Additionally,  they  had  a high  perception  of  knowledge 
retention  in  the  mixed  test,  suggesting  that  this  format  had  the  strongest  effect  among  the  three  test  formats. 


KEYWORDS 

Tablet  computer;  test  format;  test  approach-avoidance  tendency;  motivation;  information  education 


1.  INTRODUCTION 

As  a result  of  faculty  development  being  made  compulsory,  Japanese  universities  have  been  asked  to 
improve  their  classes  with  the  aim  of  promoting  student  learning  outside  of  class  (i.e.,  after-class  hours 
learning)  as  well  as  improving  student  academics  and  knowledge  retention  (Central  Education  Council,  2008). 
Furthermore,  for  information  technology  (IT),  a subject  that  became  compulsory  for  high  school  in  2003,  the 
number  of  subjects  and  teaching  content  were  revised  in  the  2013  academic  year.  Therefore,  it  is  necessary  to 
improve  information  education  classes  that  are  aimed  at  new  students  enrolled  in  IT  courses.  Blended 
learning,  which  integrates  face-to-face  lessons  with  an  e-learning  system,  has  been  proposed  as  a method  to 
resolve  this  issue  (e.g.,  Akkoyunlu  et  al.,  2008).  It  is  anticipated  that  constructing  an  environment  in  which 
students  can  access  an  e-learning  system  using  mobile  terminals  will  encourage  after-class  hours  learning,  as 
it  will  allow  student  to  engage  in  learning  anywhere  and  at  any  time. 

Moreover,  administering  tests  effectively  can  be  considered  as  a method  of  knowledge  retention  (Bloom, 
1971).  For  example,  when  teachers  give  advance  notice  of  a test,  students  are  externally  motivated  to  study, 
which  promotes  their  learning  (Kuramitsu,  1980).  Based  on  this  finding,  one  can  expect  that  asking  students 
to  take  tests  in  a mobile  environment  outside  of  class  hours  will  produce  a knowledge  retention  effect  and 
promote  after-class  hours  learning.  However,  since  learners’  perception  of  tests,  or  their  perception  of  the 
objective  of  conducting  tests  and  their  roles,  affects  their  learning  behaviors,  it  is  important  to  administer 
tests  that  take  into  account  learners’  approach-avoidance  tendencies  (Suzuki  201 1). 

Based  on  the  above  findings,  the  authors  previously  constructed  a blended  learning  environment  by 
integrating  face-to-face  classes  and  an  e-learning  system  that  utilized  mobile  terminals,  focusing  on  the  test 
approach-avoidance  tendency  in  first-year  university  information  education  classes.  Having  constructed  this 
environment,  the  authors  explored  the  motivations  toward  and  learning  effects  of  tests  conducted  using 
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mobile  terminals  outside  of  class  hours;  however,  an  analysis  of  the  validity  of  test  formats  has  not  yet  been 
conducted  (Kitazawa  et  al.,  2012).  Therefore,  this  study  experimentally  constructed  a blended  learning 
environment  by  integrating  face-to-face  classes  with  an  e-learning  system  using  mobile  terminals  in  first-year 
university  information  education  classes  and  gave  tests  using  a tablet  (iPad),  a type  of  a mobile  terminal.  This 
study  then  aimed  to  explore  the  relations  between  test  formats  and  the  test  approach-avoidance  tendency  of 
university  students  and  their  motivations  toward  tests. 


2.  METHOD 

2.1  Participants 

The  participants  were  60  Japanese  university  students  (30  men  and  30  women).  The  research  was  conducted 
on  October  14,  2012.  The  students  attended  a lecture  on  the  introductory  section  of  the  course  “Introduction 
to  Information  Science”  was  held.  The  lecture  lasted  approximately  15  minutes;  it  was  designed  to  resemble 
an  information  education  class  held  during  the  first  year  of  the  university  course. 

2.2  Administering  Tests  via  Tablets 

After  the  lecture  was  over,  the  students  took  a test  using  a tablet  terminal.  The  tests  were  distributed  during 
after-class  hours.  The  tablet  terminal  used  in  this  study  was  the  iPad  (3rd  generation),  16  GB,  Wi-Fi  model. 


Figure  1 . A mixed  test  format  of  multiple-choice  and  fill-in-the-blank  questions 

Although  there  are  various  types  of  test  formats,  this  study  focuses  on  multiple-choice  and  fill-in-the- 
blank  formats,  which  are  considered  easy  to  use  autonomously.  Each  participant  took  three  tests,  each  with  a 
different  format;  (1)  multiple-choice,  (2)  fill-in-the-blank,  and  (3)  a mixture  of  multiple-choice  and  fill-in- 
the-blank  questions  (Figure  1).  In  consideration  of  the  order  effect,  the  participants  were  divided  into  three 
groups,  with  the  orders  of  test  taking  different  for  each  group.  Regarding  the  number  of  questions,  since  five 
questions  seemed  to  be  too  few  (Kitazawa  et  al.,  2012),  10  questions  were  created  for  each  question  format  in 
this  study.  Each  of  the  three  tests  took  approximately  20  minutes  to  complete. 

2.3  Questionnaire 

To  analyze  the  university  students’  test  approach-avoidance  tendency,  the  test  approach-avoidance  tendency 
scale  (a  7-item  scale  totaling  10  questions)  developed  by  Suzuki  (2011)  was  used  as  a pre-questionnaire.  A 
post-questionnaire  survey  (a  5-item  scale  totaling  20  questions)  was  administered  to  assess  the  students’ 
“burdens  of  taking  tests,”  “willingness  to  take  tests,”  and  “self-efficacy”  as  their  motivation  regarding  tests 
(approx.  10  minutes).  The  data  were  analyzed  by  an  overall  trend  analysis  comparing  them  with  the  median 
values.  A t-test  was  also  conducted  to  analyze  the  differences  between  approach-avoidance  tendencies. 
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3.  RESULTS 

3.1  Pre-questionnaire 

A principal  component  analysis  of  the  results  of  the  test  approach-avoidance  scale  with  factors  limited  to  two 
was  conducted  with  the  data  of  58  out  of  the  60  participants,  after  excluding  two  participants  with  missing 
data.  The  numbers  of  people  with  a test-approach  tendency  or  a test-avoidance  tendency,  respectively,  were 
counted.  There  were  29  people  with  a test-approach  tendency  and  29  with  a test-avoidance  tendency. 

3.2  The  Percentage  of  Correct  Answers  in  Tests 

A two-way  factorial  analysis  of  variance  was  conducted  on  the  test-approach  and  test-avoidance  tendencies 
for  the  correct  answer  rates  of  the  tests  for  each  format  of  multiple -choice  questions  (average  of  test 
approach:  80.7,  average  of  test  avoidance:  77.9),  fill-in-the-blank  questions  (average  of  test  approach;  55.9, 
average  of  test  avoidance:  54.8),  and  mixture  (average  of  test  approach;  76.2,  average  of  test  avoidance:  74.5). 
The  results  showed  a significant  main  effect  of  test  formats  (f(2,112)  = 32.7,  p < .01).  When  Tukey’s 
multiple  comparison  test  was  conducted  thereafter,  a significant  difference  was  found  between  the  average 
values  of  multiple-choice  and  fill-in-the-blank  questions  (p  < .01),  and  between  fill-in-the-blank  and  a mix  of 
multiple-choice  and  fill-in-the-blank  questions  (p  < .01)  for  both  test  approach  tendency  and  avoidance 
tendency.  Furthermore,  no  significant  difference  was  found  between  test  approach  tendency  and  avoidance 
tendency. 

3.3  Post-questionnaire 

Table  1 shows  the  results  of  the  post-test  questionnaire  survey  on  the  burden  of  test-taking,  willingness 
toward  tests,  and  self-efficacy.  To  analyze  the  overall  trends,  comparison  analyses  were  conducted  with  a 
median  value  (3.0)  using  a t-test  for  the  average  values  of  each  item.  A significant  difference  was  found  for 
15  items  at  below  the  .01  significance  level.  A significant  difference  was  found  for  2 items  at  the  .10 
significance  level. 

Among  the  items  related  to  multiple-choice  questions,  items  such  as  (1)  “Taking  tests  with  only  multiple- 
choice  questions  is  a burden  (reversed  item)”  (average  value:  4.22)  (p  < 0.01)  had  significantly  high  values. 
This  shows  that  most  students  do  not  think  taking  tests  with  only  multiple-choice  questions  is  a burden.  On 
the  other  hand,  items  such  as  (16)  “By  taking  tests  with  only  multiple -choice  questions,  I have  become  able 
to  explain  to  others  what  I have  learned  in  this  class  (average:  2.19)  (p  < .001)”  had  significantly  low  values. 
This  shows  students  perceive  that  tests  with  only  multiple-choice  questions  do  not  promote  knowledge 
retention  and  sufficient  understanding.  Concerning  the  items  related  to  fill-in-the-blank  questions,  items  such 
as  (8)  “Tests  with  only  fill-in-the-blank  questions  lead  to  knowledge  retention  (average:  3.93)  (p  < .001)”  had 
significantly  high  values.  This  shows  that  students  that  perceive  tests  with  a fill-in-the-blank  format  achieve 
knowledge  settlement.  On  the  other  hand,  items  such  as  (2)  “Taking  tests  with  only  fill-in-the-blank 
questions  is  a burden  (reversed  item)  (average:  3.93)  (p  < .001)”  had  significantly  low  values.  This  suggests 
that  the  fill-in-the-blank  format  is  a burden  and  has  little  test  continuity. 

Furthermore,  concerning  the  post-test  survey  questionnaire  results,  the  t-test  analysis  did  not  show  a 
significant  difference  in  any  item  when  the  difference  between  test  approach  tendency  and  avoidance 
tendency  was  analyzed.  Therefore,  when  tests  were  distributed  during  after-class  hours,  using  a tablet  within 
an  autonomous  testing  situation,  among  the  three  test  types,  the  mixture  of  multiple-choice  and  fill-in-the- 
blank  questions  showed  test  continuity  and  knowledge  retention  effects,  regardless  of  whether  the  student 
tendencies  were  test-approach  or  avoidance. 


4.  CONCLUSION 

This  study  analyzed  the  effectiveness  of  test  formats  that  utilized  tablets  for  tests  in  university  information 
foundation  subjects  in  a blended  learning  environment.  Specifically,  three  types  of  test  were  created;  (1) 
multiple-choice,  (2)  fill-in-the-blank,  and  (3)  a mixture  of  multiple -choice  and  fill-in-the-blank.  An  analysis 
focused  on  university  students’  approach-avoidance  tendency  was  conducted  to  investigate  how  the  question 
formats  affected  students’  motivation  toward  the  tests.  The  results  showed  that  regardless  of  the  test 
approach-avoidance  tendency,  students  showed  a low  understanding  of  the  content  of  multiple-choice  tests. 
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even  though  they  did  not  feeling  burdened  in  taking  them.  Additionally,  for  fill-in-the-blank  tests,  although 
the  students  had  high  perceptions  of  their  knowledge  retention,  their  willingness  to  answer  the  questions  and 
the  percentage  of  questions  they  answered  correctly  were  low.  On  the  other  hand,  students  had  a high 
perception  of  continuity  and  the  percentage  of  correct  answers  was  high  on  the  test  with  a mixture  of 
multiple-choice  test  and  fill-in-the-blank  questions.  Additionally,  they  also  perceived  the  mixed  test  to  give 
them  high  knowledge  retention,  suggesting  that  this  format  had  the  strongest  effect  among  the  three. 

Future  research  should  include  exploring  a valid  number  of  questions.  Furthermore,  since  smartphones 
have  become  prevalent  as  mobile  terminals,  it  is  necessary  to  investigate  the  difference  between  smartphones 
and  the  tablets  used  in  this  study  (iPad:  3rd  generation).  In  addition,  a long-term  study  in  actual  classroom 
environments,  and  not  just  within  the  experiment  environment,  is  required. 


Table  1.  Results  of  the  post-questionnaire  (Comparison  analysis  with  medium  values) 


Items 

Avg. 

SD 

1 . Taking  tests  with  only  multiple-choice  questions  is  a burden  (reversed  item). 

4.22 

.65 

*** 

Tests  that  combine  multiple-choice  and  fill-in-the-blank  questions  lead  to  knowledge  retention. 

3.95 

.71 

*** 

8.  Tests  with  only  fill-in-the-blank  questions  lead  to  knowledge  retention. 

3.93 

.99 

*** 

14.  Tests  with  only  fill-in-the-blank  questions  make  me  feel  that  I have  sufficiently  understood  what  I 
have  learned  in  this  class. 

3.88 

.94 

*** 

4.  Tests  with  only  multiple-choice  questions  make  me  answer  the  questions  willingly. 

3.83 

.96 

*** 

10.  Tests  with  only  multiple-choice  questions  are  easy  to  answer. 

3.78 

1.04 

*** 

15.  Tests  with  both  multiple-choice  questions  and  fill-in-the-blank  questions  make  me  feel  that  I have 
sufficiently  understood  what  I have  learned  in  this  class. 

3.78 

.86 

17.  By  taking  tests  with  only  fill-in-the-blank  questions,  I have  learned  to  explain  to  others  what  I have 
learned  in  this  class. 

3.67 

1.00 

*** 

12.  Tests  with  both  multiple-choice  questions  and  fill-in-the-blank  questions  are  easy  to  answer. 

3.50 

.88 

*** 

6.  Tests  with  both  multiple-choice  questions  and  fill-in-the-blank  questions  make  me  answer  the 
questions  willingly. 

3.43 

.92 

** 

18.  By  taking  tests  with  both  multiple-choice  questions  and  fill-in-the-blank  questions,  I have  become 
able  to  explain  to  others  what  I have  learned  in  this  class. 

3.43 

.94 

** 

3.  Taking  tests  with  both  multiple-choice  and  fill-in-the-blank  questions  is  a burden  (reversed  item). 

3.17 

.94 

19.  I sufficiently  understood  what  I learned  in  this  class. 

3.12 

.94 

11.  Tests  with  only  fill-in-the-blank  questions  are  easy  to  answer. 

3.02 

1.12 

7.  Tests  with  only  multiple-choice  questions  lead  to  knowledge  retention. 

2.72 

1.07 

-1- 

13.  Tests  with  only  multiple-choice  questions  make  me  feel  that  I have  sufficiently  understood  what  I 
have  learned  in  this  class. 

2.71 

1.14 

-1- 

5.  Tests  with  only  fill-in-the-blank  questions  make  me  answer  the  questions  willingly. 

2.64 

.93 

** 

20.  I am  confident  of  the  content  I learned  in  this  class. 

2.57 

.84 

*** 

2.  Taking  tests  with  only  fill-in-the-blank  questions  is  a burden  (reversed  item). 

2.24 

.94 

*** 

16.  By  taking  tests  with  only  multiple-choice  questions,  I have  become  able  to  explain  to  others  what  I 
have  learned  in  this  class. 

2.19 

.89 

*** 

***  p < .001;  **  p < .01;  -I-  p < .10 
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ABSTRACT 

Adopting  exercise-making  and  evaluation  activities,  we  conducted  a teaching  method  of  Information  Studies  which  is  a 
teaching-training  course  subject.  We  surveyed  the  learners’  recognition  rate  of  terms  related  to  lessons  at  both  the 
beginning  and  the  end  of  lessons.  Then  we  tested  the  significance  of  the  differences  between  both  rates.  Those  results 
revealed  that  the  participants’  amount  of  knowledge  had  increased  significantly.  The  participants’  awareness  related  to 
ability  had  surveyed.  Significant  difference  tests  about  their  awareness  revealed  that  they  had  improved  significantly. 
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1.  INTRODUCTION 

Academic  education  using  manufacturing  and  evaluation  activities  to  cultivate  students’  ability  to  solve 
problems  has  been  advanced  by  the  author  (Miyaji  2009).  To  help  students  taking  college  lecture  courses, 
methods  developed  in  earlier  studies  supported  them  so  that  they  can  prepare  and  review  their  lessons 
anywhere  and  anytime  on  a personal  basis,  thereby  increasing  learning  opportunities  for  various  students 
(Miyaji  201 1).  As  a part  of  that  effort,  the  author  implemented  blended  lessons  related  to  the  introduction  of 
computer  science.  Moreover,  development  of  lectures  combining  lecture  process  notebooks,  e-learning,  and  a 
quiz  elicited  some  effects  (Miyaji  et  al.  2005).  In  Japan,  such  blended  learning  lessons  are  often  conducted. 
Mutual  reinforcement  between  lectures  and  e-learning  is  beneficial  (Miyaji  2009). 

Some  have  compared  and  reported  trial  runs  of  educational  curricula  for  teacher-training  course  subjects 
(Matsuda  1999).  However,  few  reports  have  described  educational  methods  that  have  been  actually 
introduced,  or  their  effects  (Miyaji  201 1). 

An  intensive  training  course  for  teaching  methods  used  in  information  studies  was  conducted  at  A 
University.  It  uses  blended  lessons,  in  which  students  can  learn  lesson  contents  using  lecture  slides  with  e- 
learning  outside  lessons.  We  report  herein  that  such  lessons  increased  their  amount  of  knowledge  and 
improved  their  awareness. 


2.  CONTENTS  AND  PLANS  OF  LESSONS 

A University,  the  teaching  methods  course  for  Information  Studies  is  a teacher-training  subject.  The  methods 
are  taught  in  this  subject  so  that  students  in  high  school  collect  proper  information  from  a large  amount  of 
information,  subsequently  they  can  sort  it  and  can  use  the  “information”.  For  students  in  high  school  to  solve 
problems,  they  must  be  able  to  build  their  own  knowledge  from  that  collected  information,  to  communicate  it 
properly  with  others,  and  to  transmit  it  to  others.  In  other  words,  teachers  in  Information  Studies  must  be  able 
to  have  their  students  in  high  school  think  about  large  amounts  of  information,  conduct  repeated  trial  and 
error,  and  improve  what  they  have  worked  on  and  expressed.  In  the  lessons,  the  teaching  methods  were 
taught  the  students  based  on  a textbook  (Miyaji  2004). 
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The  purposes  of  the  lessons  offered  at  A University  were  that  a student  came  to  be  able  to  do  the 
following:  to  know  teaching  goals  and  composition  of  the  subject  “information,”  to  know  the  subject  outline 
and  instruction  points,  to  master  practical  method,  to  teach  students  Information  Studies,  and  then  to  evaluate 
their  knowledge.  The  students  must  be  able  to  understand  the  teaching  method  for  information  studies  and  be 
able  to  produce  a teaching  guidance  plan  under  which  they  will  actually  conduct  classes.  Additionally,  they 
must  increase  their  necessary  capability  to  administer  and  conduct  classes,  and  to  master  the  capability  to 
solve  problems  that  can  be  predicted  to  occur  in  information  studies  lessons. 

The  information  studies  teaching  method  was  conducted  in  a closely  packed  series  of  15  lessons  of  90 
min  for  four  days.  Using  a textbook  (Miyaji  2004),  face-to-face  classes  were  conducted  by  projecting  slides 
on  the  screen.  The  contents  were  given  in  order  of  the  textbook’s  chapters  1-7.  During  the  fourth,  eighth,  and 
twelfth  lessons,  the  last  lessons  of  the  respective  days,  assignments  related  to  a learning  guidance  plan  were 
done:  the  students  were  asked  to  produce  a report  and  submit  it.  Furthermore,  by  the  twelfth  lesson  of  the 
third  day,  they  were  obligated  to  produce  their  own  learning  guidance  plan  including  PowerPoint  (Microsoft 
Corp.)  slides. 

For  the  thirteenth  lesson  of  the  fourth  day,  they  were  asked  to  make  a presentation  with  the  slides  and  to 
evaluate  it  with  other  students.  For  the  fifteenth  (final)  lesson,  the  final  examination  was  administered:  15 
students  took  the  test. 

To  intensify  the  effects  of  this  lecture  and  to  enable  the  students  to  make  a peer  assessment,  e-learning 
functions  were  added  as  follows:  (1)  learning  with  254  lecture  slides;  (2)  documents  and  templates  of  seven 
kinds  which  can  be  downloaded;  (3)  exercises  of  seven  kinds  to  submit  and  be  uploaded;  (4)  uploading 
opinions  to  a bulletin  board  and  browsing  them;  and  (5)  sending  question  mail. 

Students  were  asked  to  download  the  template  to  produce  a report.  Later,  they  submitted  the  report 
printed  on  A-4  size  paper  during  the  lesson  period. 

We  requested  that  they  make  and  submit  reports  by  the  four  exercises:  (1)  explain  the  reason  why  they 
want  to  be  an  information  studies  teacher;  (2)  produce  a learning  guidance  plan  for  an  information  studies’ 
lesson;  (3)  make  slides  for  the  learning  guidance  plan;  and  (4)  write  a presentation  manuscript  for  the 
learning  guidance  plan  and  then  present.  Only  in  terms  of  the  presentation  in  assignment  4 did  we  have  them 
make  peer  assessments. 

With  exercise  2,  participants  were  asked  to  design  concrete  lessons  in  Information  B and  Information  C 
so  that  their  students  can  solve  their  problems.  They  were  requested  to  make  a learning  guidance  plan  for 
two-unit  lessons  (100  min).  To  the  16  items,  participants  filled  out  a file  presented  on  Excel  software 
(Microsoft  Corp.). 

With  exercise  3,  based  on  exercise  2,  the  students  compiled  their  teaching  guidance  plan  to  be  executed  in 
two  unit  hours  (100  min)  on  six  slides  of  PowerPoint  software  (Microsoft  Corp.). 

With  exercise  4,  students  presented  their  own  prepared  slide  on  the  screen  in  front  of  other  students  after 
making  a presentation  manuscript.  The  allotted  time  was  five  minutes,  of  which  three  minutes  were  allotted 
for  the  presentation  and  two  minutes  were  allocated  for  questions  and  answers.  Subsequently,  they  mutually 
evaluated  what  had  been  presented.  After  the  evaluation,  they  filled  out  their  evaluations  of  others  on  the 
evaluation  sheet  and  submitted  it. 


3.  ANALYTICAL  RESULTS  AND  DISCUSSION 

After  conducting  lessons  of  teacher-training  methods  for  Information  Studies,  we  surveyed  the  recognition 
rate  to  ascertain  the  change  of  the  participants’  knowledge  amount  and  conducted  a consciousness  survey  to 
elucidate  the  extent  of  change  of  their  awareness.  We  analyzed  the  data  using  tests  of  significant  difference. 
We  explained  the  results. 

With  regard  to  50  terms  related  to  the  lecture  content  of  teacher-training  methods  for  information  studies, 
we  conducted  a pre-survey  of  the  recognition  rate  at  the  first  lesson  and  the  post-survey  at  the  fourteenth 
lesson.  All  15  participants  responded  to  both  surveys.  The  recognition  rate  responses  were  the  following 
three:  1. 1 do  not  know  it;  2. 1 do  not  know  the  details,  but  I have  heard  of  it;  3. 1 know  it. 

The  pre-average  and  post-average  rating  values  were,  respectively,  1.9  and  2.8.  From  results  of  f-tests 
corresponding  to  50  items,  we  were  able  to  infer  significant  differences.  Because  the  recognition  rate  of  the 
terms  became  higher  overall,  it  was  found  that  the  amount  of  knowledge  about  the  terms  had  increased. 
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In  terms  of  the  pre-average  and  post-average  rating  values,  the  results  of  the  f-tests  corresponding  to  each 
term  are  presented.  Results  show  significant  differences  for  45  out  of  50  terms.  From  this  result,  we  infer  that 
knowledge  about  the  terms  increased. 

In  terms  of  the  awareness  related  to  the  ability  of  30  items,  we  conducted  a pre-survey  at  the  first  lesson 
and  a post-survey  at  the  fourteenth  lesson  (Miyaji  2005).  The  rating  scale  values  were  of  following  nine 
stages:  1.  no  ability;  3.  little  ability;  5.  a little  ability;  7.  not  a little  ability;  9.  sufficient  ability. 

Respondents  to  both  the  pre-survey  and  the  post-survey  were  15.  The  results  of  f-tests  assessing  between 
the  pre-rating  scale  value  and  post  rating  scale  value  for  30  items  related  to  the  ability  showed  a significant 
difference:  awareness  related  to  ability  had  improved  as  a whole. 

The  f-tests  between  pre-average  and  post-average  rating  scale  values  of  each  item  of  awareness  related  to 
ability  were  conducted  compared,  revealing  significant  differences  for  11  items.  Furthermore,  we  found  a 
tendency  of  significant  differences  for  three  items. 

One  purpose  of  the  lecture  is  to  develop  the  ability  necessary  to  conduct  classes.  In  other  words,  it  is  to 
solve  problems  that  might  arise  when  giving  lessons  in  information  studies.  Because  “ability  to  solve 
problems”  and  the  rating  scale  values  related  to  many  other  abilities  increased,  results  suggest  that  the 
abilities  necessary  to  conduct  classes  in  information  studies  courses  should  have  also  increased. 


4.  CONCLUSION 

For  students  of  teacher-training  in  the  information  studies  at  A University,  we  requested  that  students  use  e- 
learning,  produce  a learning  guidance  plan  and  its  slides,  and  give  a presentation.  Furthermore,  we  had  them 
mutually  evaluate  the  presentations  along  with  other  students. 

Analysis  of  the  educational  information  obtained  through  conduct  of  those  classes  revealed  the  following: 

(1)  Because  the  recognition  rate  survey  of  the  terms  showed  that  the  knowledge  amount  of  the  terms 
increased  overall  and  the  knowledge  amount  of  almost  terms  increased,  it  suggests  that  the  students 
became  to  be  able  to  understand  the  teaching  method  for  information  studies  and  be  able  to  produce  a 
teaching  guidance  plan. 

(2)  Because  a survey  of  awareness  related  to  ability  showed  that  the  awareness  improved  significantly  overall, 
it  suggests  that  the  students  have  increased  their  necessary  capability  to  administer  and  conduct  classes,  and 
to  master  the  capability  to  solve  problems  that  can  be  predicted  to  occur  in  information  studies  lessons. 

The  author  appreciates  the  support  of  the  Grant-in-Aid  for  Scientific  Research,  foundation  study 
(C22500949)  provided  by  the  Ministry  of  Education,  Culture,  Sports,  Science  and  Technology,  Japan  for  this 
research.  The  author  would  like  to  express  appreciation  to  the  students  who  were  surveyed  and  who  helped 
collect  educational  information. 
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ABSTRACT 

The  study  aims  to  investigate  structural  relationships  among  teenagers’  peer  attachment,  self-control,  academic  stress, 
internet  usage  time,  and  cyber  delinquency.  The  data  source  was  the  Korea  Youth  Panel  Survey,  and  the  responses  from 
920  teenagers  in  the  12th  grade  provided  the  study  data.  Structural  equation  modeling  was  used  for  the  analysis.  The 
results  indicated  that  teenager  self-control  significantly  affected  the  internet  usage  time.  Also,  peer  attachment, 
self-control,  academic  stress,  and  internet  usage  significantly  affected  teenager  cyber  delinquency.  The  research  results 
imply  that  the  development  of  peer  attachment  and  self-control  is  important  to  reduce  cyber  delinquency. 


KEYWORDS 

Internet  usage;  cyber  delinquency;  academic  stress;  self-control 


1.  INTRODUCTION 

Internet  has  become  a critical  part  of  our  daily  lives.  Recent  research  reported  that  99.9%  of  the  Korean 
teenagers  are  using  the  internet  (Korea  Internet  & Security  Agency,  201 1).  The  issue  here  is  that  some  of  the 
teenagers  may  use  the  internet  in  an  inappropriate  manner,  which  results  in  cyber  delinquency,  for  example, 
distributing  false  or  misleading  information,  sharing  software  illegally,  and  being  aggressive  ignoring 
netiquettes.  This  study  tried  to  understand  the  phenomenon  of  teenager  cyber  delinquency  by  investigating 
predicting  variables  for  online  misbehaviors.  From  the  theoretical  perspective,  Hirschi  (1969)  proposed 
‘social  bonding  theory’,  focusing  on  attachment,  commitment,  involvement,  and  belief  as  critical  factors  for 
reducing  teenager  delinquency.  Bowlby  (1982)  also  claimed  that  attachment,  especially  with  peers,  plays  an 
important  role  for  teenagers  to  shape  their  behavior.  On  the  other  hand,  Gottfredson  and  Hirschi  (1990) 
suggested  ‘general  theory’  to  explain  self-control  as  a predicting  factor  for  delinquency,  and  the  significant 
effect  of  self-control  on  delinquency  has  been  echoed  by  previous  studies  such  as  Pratt  and  Cullen  (2000), 
and  Higgins  (2004).  ‘General  strain  theory’  by  Agnew  (1992)  and  ‘opportunity  theory’  by  Osgood,  Wilson, 
O’Malley,  Bachman,  and  Johnston  (1996)  proposed,  respectively,  that  stress  level  and  the  exposure  to  the 
opportunity  would  affect  the  degree  of  delinquency. 

Based  on  the  extensive  literature  review,  researchers  suggested  peer  attachment,  self-control,  academic 
stress,  internet  usage  time  as  tentative  predictors  for  cyber  delinquency.  Prior  research  in  the  context  of 
teenager  cyber  delinquency  indicated  that  peer  attachment  and  self-control  negatively  predicted  the  internet 
usage  time  and  cyber  delinquency  (e.g.  Higgins,  2004;  Mesch,  2002).  Leung  (2007)  claimed  that  teenagers 
are  likely  to  reduce  their  stress  level  by  surfing  the  internet  and  interacting  with  peers  online,  which  is  closely 
related  to  the  hours  spent  online.  Lastly,  Ybarra  (2004)  reported  that  the  internet  usage  time  significantly 
predicted  cyber  delinquency. 

Hence,  the  researchers  investigated  the  causal  relationships  among  the  suggested  variables.  Research 
questions  are  as  follows:  (1)  Do  teenagers’  peer  attachment,  self-control,  and  academic  stress  level  predict 
the  internet  usage  time?  (2)  Do  teenagers’  peer  attachment,  self-control,  academic  stress  level  and  the  internet 
usage  time  predict  cyber  delinquency? 
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2.  METHODOLOGY 

The  data  source  was  the  Korea  Youth  Panel  Survey,  and  the  responses  from  920  teenagers  in  the  12th  grade 
provided  the  study  data.  Survey  participants  were  sampled  by  stratified  random  sampling  in  terms  of  the 
province.  Demographically,  58%  of  the  respondents  were  male  and  42%  female. 

Survey  questionnaire  for  the  variables,  peer  attachment,  self-control,  academic  stress,  internet  usage  time, 
and  cyber  delinquency  was  developed  by  Korea  National  Youth  Policy  Institute,  and  develop  and  reviewed 
by  academic  and  field  experts.  Regarding  peer  attachment,  self-control,  and  academic  stress,  Cronbach’s  a 
from  the  study  data  ranged  from  .73  to  .81,  and  used  5 point  Likert  scale.  Internet  usage  time  was  measured 
by  hours  spent  for  the  internet  per  day,  while  cyber  delinquency  was  measured  by  the  frequency  of  behaviors 
relevant  to  the  indicators  of  cyber  delinquency  during  the  recent  12  months. 

Structural  equation  modeling  was  used  for  the  analysis,  and  SPSS  and  AMOS  were  used. 


3.  RESULTS 


Descriptive  analysis  indicated  there  were  significant  correlations  among  all  the  variables  at  the  significance 
level  of  .05,  and  the  assumption  of  normal  distribution  was  met.  As  factor  analysis  confirmed  that  each 
variable  is  unidimensional,  item  parceling  was  adopted  to  minimize  measurement  errors  in  stmctural 
equation  modeling. 

As  a result  of  maximum  likelihood  estimation  and  stmctural  model  testing,  the  relationships  between  peer 
attachment  and  internet  usage,  and  between  academic  stress  and  internet  usage  were  insignificant.  Therefore 
the  modified  model  was  tested  again,  and  it  provided  a good  fit  to  the  data  (e.g.,  TLI=  .990;  CFI=  .995; 
RMSEA=  .019  (.000-.040)).  Figure  1 presents  the  modified  model  with  significant  path  coefficients. 


Figure  1 . Modified  model  with  standardized  path  coefficients 


4.  CONCLUSION 

The  researchers  tried  to  provide  more  elaborated  understanding  on  teenager  cyber  delinquency.  Unlike  the 
results  from  prior  studies,  peer  attachment  and  academic  stress  did  not  affected  internet  usage  time.  It  is 
likely  that  peer  attachment  these  days  is  maintained  or  even  increased  through  online  social  network  services. 
Academic  stress  is  another  variable  reflecting  research  context.  As  Korean  teenagers,  especially  12*  graders, 
are  highly  focused  on  college  entrance  exam,  they  may  try  to  reduce  the  hours  spent  online  regardless  of  their 
stress  level. 

Overall,  the  results  provide  implications  on  which  aspects  we  should  focus  on  in  order  to  intervene 
teenager  cyber  delinquency.  Also,  the  results  indicate  that  we  should  explore  peer  attachment  from  a new 
perspective,  since  a variety  of  social  network  services  are  influencing  the  perceived  level  of  attachment  with 
peers  online  and  offline  as  well.  Further  studies  are  required  to  investigate  more  elaborated  model  with  social 
and  environmental  variables  as  predictors. 
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ABSTRACT 

Through  the  previous  detection  of  existing  learning  styles  in  a classroom,  a Virtual  Learning  Environment  (VLE)  has 
been  designed  for  students  of  several  Engineering  degrees,  using  the  Learning  Management  System  (LMS)  utilized  in  the 
University  of  Jaen,  ILIAS.  Learning  styles  of  three  different  Knowledge  Areas;  Chemical  Engineering,  Materials  Science 
and  Environment  Technology,  have  been  analyzed.  Erom  learning  styles  detected  in  the  classroom,  the  introduction  of 
optional  and  compulsory  activities  to  be  developed  by  the  student  permits  to  develop  the  VLE.  In  this  “optionality”,  the 
student  can  choose  the  activities  more  suitable  to  his  learning  style.  As  a consequence  of  the  implementation  of  the  virtual 
learning  style,  the  assessment  results  must  improve,  as  well  as  the  students’  motivation  because  they  are  selecting  their 
particular  learning  way. 


KEYWORDS 

Learning  styles,  e-leaming,  virtual  learning  environment 


1.  INTRODUCTION 

In  the  context  of  the  European  Higher  Education  Area  (EHEA)  an  important  change  in  the  way  of  carrying 
out  the  teaching-learning  process  is  necessary.  So  that  the  virtual  learning  plays  a primordial  role  making 
possible  the  following  of  the  subject  in  case  of  the  student  cannot  attend  to  the  class,  at  the  time  as  it  involves 
a valuable  complement  during  the  class. 

Learning  styles  represent  the  personal  way  that  each  person  has  in  order  to  acquire  learning.  The  detection 
of  them  in  the  class  is  going  to  allow  us  to  design  a virtual  learning  environment  where  the  considered 
activities  were  oriented  to  the  detected  styles,  basically.  This  fact  has  to  have  positive  effects  in  the 
assessment  outcomes  of  the  students  and  in  the  process  of  knowledge  acquisition  [1]. 


2.  DEVELOP 

The  learning  styles  analysis  has  been  realized  in  three  Engineering  subjects  belonging  to  the  Knowledge 
Areas  of  Chemical  Engineering,  Material  Science  and  Environment  Technology  in  the  Higher  Polytechnical 
School  of  Linares,  corresponding  to  the  degrees  of  Chemical  Engineering,  Mechanical  Engineering  and 
Mining  Engineering,  respectively. 

The  learning  management  system  ILIAS  is  available  for  all  students  registered  for  this  University,  in  this 
VLE  is  used  to  distribute  contents  of  the  subjects  and  to  propose  the  different  activities.  It  is  used  as  a mean 
of  on-line  collection  of  the  proposed  activities,  as  a mean  of  evaluation  through  on-line  test  and  as  a mean  of 
communication  between  students  and  lecturer  and  among  students.  Both  a complement  of  classroom  teaching 
and  in  distance  learning  course,  the  use  of  this  learning  management  system  supposes  a fundamental  tool  to 
develop  the  VLEs. 
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2.1  Detection 


The  detection  process  of  learning  styles  has  been  done  using  the  Honey- Alonso  Learning  Styles  Survey  [2] 
based  on  the  learning  styles  classification  of  Honey  and  Mumford  [3]  where  four  styles  have  been  defined: 
activist,  reflector,  theorist  and  pragmatic. 

The  Table  1 shows  the  activities  to  which  the  student  will  react  positively  from  the  general  behavior 
expected  for  each  type  of  learning  styles  [4],  considering  the  personal  characteristics  of  the  student.  Besides, 
the  list  of  proposed  activities  in  the  virtual  learning  environment  designed. 

Table  1.  Suitable  activities  for  each  learning  style  and  proposed  activities 


Learning 

Style 


Activity  types 


React  positively  to: 


Activities  that 

■ Action  learning 

demand  a 

■ Business  game  simulations 

prompt 

■ Job  rotation 

solution  (e.g.. 

■ Discussion  in  small  groups 

solving 

■ Role  playing 

problems) 

■ Training  others 

■ Out  door  activities 

Activities  to 

■ E-leaming 

>3 

think  before 

■ Learning  reviews 

1 

acting  (e.g.. 

■ Listening  to  lectures  or 

developing  a 

presentations 

project) 

■ Observing  role  plays 

■ Reading 

■ Self-study/self-directed 

learning 

Activities  with 

■ Analytical  reviewing 

clear  objectives 

■ Exercises  with  a right  answer 

and  relative  to 

■ Listening  to  lectures 

§ 

3 

a schema  (e.g. 

■ Self-study/self-directed 

searching  of 

learning 

information  or 

■ Solo  exercises 

interpretation 

■ Watching  ‘talking  head’ 

of  a theory) 

videos 

». 


Activities  with 
practical 
applications 
(e.g.  laboratory 
sessions). 


■ Action  learning 

■ Discussion  about  work 
problems  in  the  organisation 

■ Discussion  in  small  groups 

■ Problem-solving  workshops 

■ Group  work  with  tasks  where 
learning  is  applied 

■ Project  work 


Proposed  activities 

• Provide  the  student  specific  information:  facts, 
data,  results 

• Provide  industry  media  abstract  concepts: 
principles,  theories,  mathematical  models 

• Methods  of  resolution  of  practical  problems, 
with  an  emphasis  on  understanding  the  same 

• Observation  of  surroundings,  empirical 
experimentation,  attention  to  detail. 

• Provide  concrete  examples  of  the  phenomena 
the  theory  describes  or  predicts 

• Application  of  theoretical  models 

• Using  diagrams,  graphics,  pictures,  sketches, 
films;  before,  during  and  after  the  presentation 
of  verbal  material 

• Provide  demonstrations. 

• Use  computer-assisted  instruction 

• Assign  some  drill  exercises  to  provide  practice 
in  the  basic  methods  being  taught  but  do  not 
overdo  them  and  some  open-ended  problems 

• Motivate  learning.  Talks  and  conferences  with 
graduates,  which  allow  you  to  view  the 
implementation  of  studies. 

• Theoretical  models  applications 

• Practical  troubleshooting  methods  with  an 
emphasis  on  understanding  the  same 

• Empirical  experimentation 

• Provide  intervals  for  students  to  think  about 
what  they  have  been  told. 

• Assign  some  drill  exercises  to  provide  practice 
in  the  basic  methods  being  taught  but  do  not 
overdo  them  and  some  open-ended  problems 

• Given  students  the  option  of  cooperating  on 
homework  assignments  to  the  greatest  possible 
extent. 

• Theoretical  models  applications 


2.2  Results 

After  carrying  out  the  analysis  of  results  in  each  one  of  the  classrooms,  it  has  been  observed  that  although 
every  learning  styles  are  present,  only  two  styles  prevails  over  the  rest;  the  theorist  and  reflector  styles. 
Figure  1 shows  the  distribution  of  different  learning  styles  in  a classroom  corresponding  to  the  Knowledge 
Area  of  Environmental  Technology,  it  is  possible  to  see  some  isolated  cases  of  predominance  of  activist  and 
pragmatic  styles,  but  theorist  and  reflector  styles  are  more  important.  This  behavior  also  has  been  observed  in 
the  others  engineering  class  studied  (Mechanical,  Chemical  and  Mining  Engineering). 
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Figure  1 . Distribution  of  different  learning  styles  in  a classroom 

When  the  comparison  of  the  learning  styles  is  realized  among  Knowledge  Areas  the  reflector  and  theorist 
styles  stand  out  the  others  styles  (Figure  2).  In  case  of  Chemical  Engineering  students  the  theorist  style  is 
more  important,  while  in  Materials  Science  and  Environmental  Technology  the  reflector  style  prevails. 
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■ Activist 

■ Reflector 

■ Theorist 

■ Pragmatist 


Eigure  2.  Distribution  of  different  learning  styles  in  each  analyzed  knowledge  area 

Considering  all  evaluated  engineering  students  in  the  comparison,  the  reflector  styles  is  the  more 
significant  style  followed  by  the  theorist  style  and  the  activist  is  the  style  with  less  representation.  This 
behavior  is  shown  in  Eigure  3. 

Pragmatist 
Theorist 
Reflector 
Activist 

0 10  20 
% Contribution 

Figure  3.  Overall  distribution  of  different  learning  styles 


3.  CONCLUSIONS 

The  four  learning  styles  defined  by  Honey  and  Mumford  are  present  in  the  evaluated  classroom,  however  and 
the  opposite  of  what  it  is  expected  for  Engineering  students  [5]  the  more  significant  styles  are  the  reflector 
and  the  theorist. 
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Making  the  comparison  among  Knowledge  Areas,  in  Materials  Science  and  Environmental  Technology 
the  reflector  style  is  the  more  widely  represented,  while  the  theorist  style  prevails  in  case  of  Chemical 
Engineering  students. 

In  the  Virtual  Learning  Environment  designed,  the  on-line  teaching  using  the  Learning  Management 
System,  ILIAS  perform  a fundamental  role  as  a mean  to  show  contents  and  activities.  The  proposed  activities 
must  be  oriented  to  the  four  learning  styles,  but  the  activities  that  involve  thinking  before  acting  and  activities 
with  clear  objectives,  more  connected  with  the  reflector  and  the  theorist  styles,  must  be  more  significant. 

Considering  the  characteristics  of  the  taught  subjects,  it  is  necessary  to  raise  mandatory  activities  closer  to 
activist  and  pragmatic  styles,  nevertheless  in  the  designed  VLE  the  possibility  of  choosing  others  optional 
activities,  adapted  to  the  learning  special  characteristics  of  the  student,  will  allow  improving  the  learning  and 
assessment  results,  unavoidably. 

Though  the  development  of  a Virtual  Learning  Environment  involves  an  increase  of  the  teacher  work, 
also  implicates  an  improvement  of  the  motivation  and  the  involvement  of  the  student  in  their  own  teaching - 
learning  process.  The  use  of  ILIAS  makes  possible  that  in  this  personalized  model  allow  can  be  translated  to 
any  other  type  of  university  teaching,  both  to  distance  learning  and  present  learning  in  the  classroom. 
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ABSTRACT 

This  paper  presents  about  online  course  for  librarians  of  Technical  and  Vocational  Education  and  Training  institutions. 
Target  group  was  librarians  who  work  in  rural  regions  of  the  country.  The  libraries  plays  important  role  in  population’s 
education  development.  The  general  digital  literacy  of  population  is  increasing  very  fast.  Relating  with  this  background 
we  need  to  give  corresponding  attention  to  e-library.  One  of  the  basic  factors  for  e-library  development  is  human  resource 
sector.  According  by  appeared  request  we  organized  online  course  for  librarians.  Main  target  was  to  give  opportunity 
learns  and  increase  computer  skills  on  working  place  in  regional  place.  Main  sponsor  for  this  project  was  Millennium 
Challenge  Account-Mongolia.  It  was  first,  distance  and  full  online  course  whose  target  group  was  librarians  from  rural 
regions  of  the  country. 


KEYWORDS 

E-learning,  online  course,  librarian,  distance  learning,  online  learning. 


1.  INTRODUCTION 

Mongolia  is  located  in  the  center  of  Asia,  which  is  19th  biggest  country  by  the  ground  size  in  the  world. 
Mongolia  has  about  three  million  populations.  Big  distance  between  capital  cities  and  small  towns  is  a main 
problem  in  continuously  education  for  adults.  The  libraries  in  rural  regions  of  country  is  playing  main  role  for 
education.  “The  Millennium  Challenge  Account-Mongolia  (MCA-Mongolia),  funded  through  a grant  from 
the  Government  of  the  United  States  of  America,  was  established  to  support  economic  development  and  the 
reduction  of  poverty”  noted  in  the  brochure  of  project(2012).  In  this  course  20  participants  attended  from 
different  aimakas  and  cities  (Ulaanbaatar-4,  Central  region-3,  West  region  4,  Region  “Khangai”-3,  Region 
“Gobi”-5  and  Eastern  region- 1). Course  content  was  focused  on  information  and  communication  technology 
and  e-library.  The  librarians  had  big  opportunities  to  attend  in  advanced  training  course  from  working  place. 
It  was  first  distance  and  online  learning  course  for  all  participants.  Online  service  of  library  is  very  important 
sector  in  other  countries;  too  for  instance  we  refer  to  (Alii,  A.  S.  Mcharazo,  2000;  Britt  , 2004;  Technical 
Report,  2013). 


2.  DISTANCE  ONLINE  COURSE  EOR  LIBRARIAN 

National  Learning  Resource  Center  (NLRC)  at  the  Mongolian-Korean  Polytechnic  College  was  established 
by  Vocational  Education  and  training  (VET)  project,  MCA-Mongolia  (Brochure  of  MCA-Mongolia,2013). 
The  project  founded  many  activities  and  sub  projects  which  oriented  to  development  of  information  and 
communication  technology  for  education  sector.  Inside  the  project  activity,  we  planned  two  piloting  online 
courses.  One  of  these  courses  was  focused  on  the  target  group  of  librarians. 

We  conducted  training  needs  survey  from  40  librarians  who  are  from36TVET  institutions.  The  63%  of 
the  librarians’  background  were  professional  degree  in  librarian  field.  The  main  criterion  of  the  selection 
participants  was:  some  technical  requirements  like  internet  connection,  computer  and  other.  Additionally  we 
requested  basic  ICT  knowledge:  email  using  skill,  typing  on  computer  and  working  skills  with  office 
applications.  The  subject  of  online  course  was:  “Usage  of  library  and  Information  and  communication 
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Technology”.  It  was  tutored  by  three  tutors  and  has  a consult  expert.  Learning  management  system  was 
supported  by  Mongolian  e-Knowledge  campus. 

Course  content  was  included  following  subjects: 

• Influence  of  high  technology  development  in  library  and  information  systems,  effective  usage  of  web 
2.0  technology  in  library,  e-Library,  its  services,  trends  of  library  development  in  electronic  version,  e- 
resources  and  management,  online  and  offline  catalog 

• The  management  of  copyright  problems  in  library  services.  Introduction  about  a Union  Catalog  of 
Library  -World  Cat,  creation  and  organization  of  electronic  readings.  Introduction  of  EBSCO  system 
and  Introduction  about  worldwide  famous  libraries  and  their  experiences 

Course  content  was  developed  by  library  experts.  Whole  content  covered  40  hours  of  online  course. 
Reflection  and  learned  lessons: 

• This  first  online  course  was  good  opportunity  for  participants  to  develop  professional  education  on  the 
working  places  using  internet  and  online  learning  management  system 

• The  video  conference  done  between  Mongolian  rural  regional  towns  and  Europe,  successfully  first  time. 
This  event  very  engaged  organizers  and  participants. 

• Computer  skill  of  participants  was  very  different.  Sometimes  it  was  problem  in  management  of  course 
schedule  and  synchronized  activities  with  participants 

• The  course  content  has  to  improve  in  next  course.  Get  experience  sustainable  and  flexible  e-learning 
content  for  librarians  in  the  future. 

After  online  course  we  did  summative  evaluation  using  checklist.  Chart  1 and  Chart  2 are  showed  some 
result  from  summative  evaluation.  Question:  “Are  you  satisfied  on  the  course  content?” 


Very  satisfied  Satisfied  Dissatisfied  Neutral 


Chart  1.  Satisfaction  of  learners  (course  content) 

The  90%  of  participants  answered  “very  satisfied”  on  course  content.  Question:  “Quality  of  reading 
materials  and  level  of  exercises  evaluation  and  barriers  in  learning”.  Erom  participants  66.6%  had 
distinguished  problems  with  learning. 
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■ Langauge  and  conqiuier  skills 
problem 

■ Schedule  and  content  was  not 
good  adapted 

■ Not  enough  experiences  widt 
distance  learning 

■ Learning  sigiport  was  not  enough 

■ CollaberatioQ  environment  was 
not  good 


Chart  2.  Reading  materials,  exercizes  and  barriers  in  learning 


3.  CONCLUSION 

This  project  was  first  online  course  for  “librarians”  as  target  group.  The  librarians  who  are  working  in  rural 
region  of  country  had  opportunity  to  attend  to  professional  advanced  course  from  working  places.  Aim  of 
project  was  suggested  with  mission  of  MCA-Mongolia.  A professional  advanced  training  course  for 
librarians  is  organized  not  so  often,  especially  from  distance.  In  this  case  our  project  was  reflected  very 
positive  from  participant  side.  By  summative  evaluation  it  proofed  by  90%  satisfaction  result. 

Online  course  organized  well  and  the  satisfaction  of  participants  was  very  positive.  But  some  participants 
were  dropped  out  and  some  participants  could  not  adapt  in  distance  and  online  learning.  Summative 
evaluation  after  course  useful  for  us  to  recognize  elements  which  we  need  to  improve  in  next  same  courses. 
For  example,  evaluation  result  showed  that  we  need  to  redesign  course  content  and  have  to  support 
participants  to  adapt  in  distance  and  online  learning.  Have  to  improve  collaborative  learning  environment  and 
learning  support  helps  and  hints  in  course. 

It  was  big  challenge  for  us  MCA-Mongolia.  It  was  big  opportunity  to  collect  online  and  distance  learning 
experiences  for  participants.  All  participants  first  time  attended  to  online  course  which  targeted  to  librarian. 
Every  participant  confirmed  own  satisfaction  by  summative  evaluation  checklist.  The  organizers,  the 
participants  and  sponsors,  each  involved  interested  part  learned  a lot  of  from  this  distance  online  course  case. 
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ABSTRACT 

A computerized  attention-training  game  system  has  been  developed  to  support  attention  training  for  school-aged  children. 
The  present  system  offers  various  types  of  computer  games  that  provide  training  in  different  aspects  of  attention,  such  as 
selective  attention,  sustained  attention,  and  divided  attention.  The  N-tier  architecture  of  the  Web-based  applications  was 
applied  for  the  development  of  this  system.  Children  can  access  the  system  to  play  interactive  games  in  a Web-based 
environment  by  using  smart  phones,  tablet  computers,  or  personal  computers.  A complete  set  of  training  information  is 
collected  in  which  the  system  provides  a training  portfolio  mechanism  to  record  the  training  progress  and  performance  of 
the  player.  Children  may  benefit  with  the  challenging  features  of  attention  demands  during  game-playing  and  the 
observer  may  gain  detailed  information  of  the  training  process.  The  game  scenario  helps  motivate  children  to  practice 
attention  tasks  with  interactive  play  and  immediate  feedback. 


KEYWORDS 

Computer  games.  Attention  training.  Training  portfolio.  Attention  assessment 


1.  INTRODUCTION 

Attention  is  very  important  to  the  progress  of  children's  memory  and  learning  skills  because  it  affects  the 
development  of  cognition.  Therefore,  attention  training  is  considered  an  effective  approach  to  improving  the 
cognitive  development  of  school-aged  children.  The  major  principle  is  that  repeated  practice  of  attention 
tasks  can  produce  adaptations  in  the  underlying  neuroanatomical  networks  linked  to  these  processes  and  can 
improve  the  attention  performance  (Kerns,  et  al,  1999;  Posner  and  Raichle,  1994).  Attention  training  employs 
different  kinds  of  cognitive-behavioral  tasks  (e.g.,  self-monitoring,  self-reinforcement,  neurofeedback  or 
biofeedback,  and  cognitive  training  methods)  to  train  various  types  of  attentions  (e.g.,  selective,  sustained, 
and  divided  attention)  and  to  improve  concentration  in  different  cognitive  tasks  (Levesque  et  al,  2006; 
Sohlberg  and  Mateer,  1989;  Zoefel  et  al,  2011).  Klimkeit  et  al  (2004)  investigated  the  development  of 
attention  and  executive  functions  on  normal  children  aged  between  7-12  years  old  (n  = 40).  Their  study 
results  revealed  that  the  largest  improvement  in  selective  attention  and  executive  functions  occurred  between 
the  ages  of  8-10  years  and  rose  to  a plateau  in  performance  between  10-12  years  of  age. 

Computer-based  training  uses  computer  programs  to  coach  learners  to  achieve  better  performance  by 
employing  cognitive  strategies  and  processing.  There  are  several  potential  benefits  of  the  computer-based 
training  approach,  including  the  ability  to  train  anywhere  and  anytime,  cost-effective,  and  easy  control  of  the 
training.  In  addition,  computer  and  video  games  also  facilitate  participation  and  practice  in  a personalized 
training  environment  towards  a successful  experience,  which  may  be  impossible  in  real  life  (Verenikina  et  al, 
2003).  Using  computer  games  as  training  methods  can  have  several  advantages  due  to  the  interactive 
elements  of  computer  games,  along  with  the  computer's  function  to  present  multiple  types  of  media. 
Therefore,  games  were  utilized  as  an  interesting  and  motivating  environment  of  attention  training,  and  may 
offer  different  kinds  of  simulated  situations  (such  as  iterated  practices  of  attention-cognitive  operations)  to 
improve  the  cognitive  performance  during  attention  training. 
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The  purpose  of  this  paper  presents  the  computerized  attention-training  game  system,  which  is  targeted  at 
school-aged  children  to  improve  the  performances  of  the  visual  attention  and  working  memory.  The  game- 
based  attention  training  system  can  provide  children  not  only  with  fun  scenario  games  but  also  with  exercise 
to  practice  attention  training. 


2.  DESIGN  AND  IMPLEMENTATION 

The  architecture  of  the  present  system  consists  of  five  modules:  a user  interface  module,  user  profile 
management  module,  game  management  module,  training  portfolio  module,  and  performance  management 
module.  The  user  interface  module  includes  three  different  interactive  interfaces  for  children,  instructors,  and 
administrators.  The  user  profile  management  module  mainly  stores  the  user's  basic  information  such  as  name, 
age,  and  education.  The  game  management  module  is  used  for  the  management  and  integration  of  different 
attention  training  games.  The  training  portfolio  module  provides  users  with  the  information  of  training 
progress.  Users  can  query  detailed  information  of  their  individual  responses  for  each  game.  Instructors  or 
administrators  also  can  analyze  the  progress  trend  regarding  an  individual  user  or  a group  of  participants 
from  the  data  extracted  from  this  module.  The  performance  management  module  computes  and  manages  the 
results  of  individual  game  playing.  The  module  can  provide  users  with  the  scores  of  a single  game  or  the 
whole  game. 

Five  games  have  been  developed  in  the  system  (see  Figure  1)  and  a series  of  attention-training  tasks,  such 
as  selective  attention,  sustained  attention,  and  divided  attention,  are  involved  in  each  game.  For  selective 
attention  training  in  the  game,  the  visual  stimuli  included  a group  of  animals,  such  as  sheep,  cows,  chickens, 
ducks,  and  pigs,  which  are  placed  in  a farm  where  the  player  has  to  help  the  farmer  identify  the  chicken  from 
the  other  animals  (i.e.,  leading  the  chicken  out  apart  from  the  other  animals  in  the  farm).  These  animals  might 
have  similar  features  in  color  and  shape  (such  as  chickens  and  ducks)  or  different  features  (such  as  chickens 
and  cows)  to  distinguish  the  level  of  selective  difficulties  in  the  same  game  (see  Figure2).  Between  different 
game  levels,  animals  can  be  maintained  in  the  farm  in  a lower  game  level  or  moved  to  a different  spot  in  a 
higher  level.  The  moving  targets  increase  the  difficulty  for  the  player  to  capture  the  target  animal  in  the  farm. 
Similar  to  a common  video/computer  game,  a visual  and  audio  feedback  will  appear  showing  correct,  wrong, 
or  missing  responses  after  each  response  during  the  game. 


Figure  1.  Game  menu  Figure  2.  Chicken  catcher 

This  system  also  provides  a training  portfolio  mechanism  to  manage  players'  training  progress.  School 
children  (players)  or  teachers  can  access  the  record  in  the  training  games,  such  as  frequencies,  scores, 
response  types  (correct,  error,  or  omission),  and  play  time  for  each  program,  to  keep  track  of  their  session 
performances  and  training  progresses  through  sessions. 

The  implementation  of  the  system  includes  JavaScript  and  PHP  as  script  languages.  Flash  as  an  authoring 
tool,  Apache  as  a web  server,  and  MySQL  as  a database.  Training  progresses  and  session  performances  are 
recorded  in  MySQL  database.  The  database  contains  three  kinds  of  relational  tables  including  user  profiles, 
portfolio  data,  and  result  data. 


487 


ISBN:  978-972-8939-88-5  © 2013  lADIS 


3.  CONCLUSION 

This  study  proposed  a computerized  attention-training  game  system  to  train  attention-related  function  in 
children.  The  Web-based  applications  own  the  merit  of  "anytime  and  anywhere  accessibility"  for  attention 
training  practice  at  home,  school,  and  clinical  settings.  To  evaluate  the  performance  of  attention  training  in 
the  present  system,  we  are  currently  conducting  experimental  sessions  of  attention  training  in  school-aged 
children  to  verify  the  efficacy  of  the  system. 
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ABSTRACT 

The  purpose  of  this  study  was  to  develop  an  attention  sequential  mining  mechanism  for  investigating  the  sequential 
patterns  of  children's  visual  scanning  process  in  a computerized  cancellation  test.  Participants  had  to  locate  and  cancel  the 
target  amongst  other  non-targets  in  a structured  form,  and  a random  form  with  Chinese  stimuli.  Twenty-three  children 
aged  between  10-12  years  were  recruited  and  took  part  in  this  experiment.  Sequential  pattern  analysis  for  stimulus 
cancellation  revealed  that  the  structured  form  has  a visual  guiding  effect  producing  a maximum  of  18-sequence  in  a 
horizontal  scanning  pattern.  The  sequential  pattern  of  the  random  form,  showed  no  guiding  effect,  was  vertical  and 
yielded  only  2-  to  4-sequence  patterns.  The  results  demonstrated  that  the  organization  of  stimulus  layouts  (i.e.,  structured 
or  random  forms)  might  have  implicit  visual  guiding  effects  in  visual  scanning. 


KEYWORDS 

Cancellation  test.  Sequential  pattern  mining.  Visual  scanning,  Computerized  assessment 


1.  COMPUTERIZED  CANCELLATION  TESTS 

Visuospatial  attention  is  how  someone  attends  to  the  visual  stimuli  in  a specific  space.  Cancellation  tests 
have  been  largely  used  to  evaluate  the  performance  of  visuospatial  attention  in  education  and  medical 
settings.  Individuals  are  required  to  cancel  the  target  scattered  randomly  throughout  the  structured  or  random 
array.  A computerized  cancellation  test  system  can  trace  and  record  the  sequence  of  cancellation  mapping 
onto  the  test  screen.  The  response  path  would  represent  a visual  scanning  pattern  in  a series  of  lines 
connecting  the  positions  by  order  of  the  cancelled  stimuli. 

Sequential  pattern  mining  is  one  of  data  mining  technologies  concerned  with  interesting  sequential 
patterns  based  on  time  series  among  a large  database  (Chand  et  al,  2012).  With  huge  amounts  of  data 
continuously  being  collected  and  stored  by  a computerized  cancellation  test  system,  sequential  pattern  mining 
can  be  used  to  discover  the  potential  interesting  sequential  patterns  of  the  visual  scanning  paths.  In  this  study, 
a K-sequence  is  an  ordered  list  of  items  f (i  = 1~  k)  such  that  S = <f,  I2,  4>.  Sequence  5 in  a cancellation 

task  that  contains  k sequences  of  cancellations  (i.e.,  k successive  responses)  is  called  a frequent  sequential 
pattern  if  it  occurs  more  than  the  minimum  support  threshold  value. 

Previous  studies  suggested  different  findings  regarding  the  patterns  of  visual  searching  in  visuospatial 
tasks.  Abed  (1991)  explored  eye  movement  among  subjects  with  different  cultural  backgrounds.  The  study 
showed  that  East  Asian  subjects  used  more  vertical  than  horizontal  movements  comparing  with  American  or 
Middle  Eastern  subjects.  Woodword  (1972)  addressed  that  the  distance  between  the  stimuli  would  influence 
the  scanning  pattern  and  lead  the  individual  to  search  for  the  next  closest  target.  A previous  study  found  that 
a structured  form  with  the  Chinese  stimulus  elicited  a more  organized  visual  scanning  pattern,  but  a 
randomized  form  did  not  (Wang  et  al,  2006).  However,  the  traces  of  visual  scanning  in  both  forms  were  not 
quantitative  data  and  thus,  cannot  be  compared  statistically  to  examine  the  differences  between  scanning 
patterns. 

Therefore,  the  purpose  of  this  study  was  to  develop  an  attention  sequential  mining  mechanism  to  discover 
the  sequential  patterns  of  children's  visual  scanning  process  in  order  to  validate  the  discrepancies  of  scanning 
patterns  in  different  layouts. 
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2.  EXPERIMENT 


2.1  Subjects 

A total  of  23  participants  (9  boys,  mean  age  = 11.33  years;  14  girls,  mean  age  = 11.43  years)  participated  in 
the  experiment.  All  recruited  participants  were  free  of  any  physical  illness,  neurological  deficits,  visual  or 
motor  problem,  or  perceptual  disorders.  Participants  were  instructed  and  allowed  to  practice  until  they 
understood  the  procedure  before  starting  the  test. 

2.2  Materials 

Two  different  stimulus  layouts  (i.e.,  structural  array  and  random  array)  with  the  Chinese  stimulus  were  used 
in  this  experiment.  The  target  on  the  Chinese  test  was  the  radical  "H"  (meaning  "sun").  Other  Chinese  radical, 
such  as  and  were  used  as  non-targets  in  the  test.  In  the  structured  form,  the  spacing  of  any  two 
adjacent  stimuli  in  a row  or  column  was  equivalent,  but  for  the  random  form,  the  spacing  of  any  two  adjacent 
stimuli  varied. 

To  mine  the  frequent  sequences  for  visual  scanning  patterns,  a user-specified  minimum  support  was 
required  as  the  threshold  for  discovering  all  sequences  having  a frequency  greater  than  the  minimum 
threshold.  Based  on  the  testing  result,  a minimum  support  threshold  was  set  as  0.2  in  this  study. 


3.  RESULTS  AND  DISCUSSIONS 


A total  of  2650  records  of  participants'  responses  were  analyzed:  1322  records  for  the  structured  form  and 
1328  records  for  the  random  form.  The  results  showed  that  the  structured  form  has  a visual  guiding  effect, 
and  a maximum  of  18-sequence  horizontal  pattern  was  found  (see  Figure  1).  Most  horizontal  patterns  were 
left-right  or  right-left  tracing  as  an  S-shape  with  top-down  order  in  the  structured  form.  However,  the  random 
form  showed  no  guiding  effect  and  only  a maximum  of  4-sequence  vertical  pattern  was  found  (see  Figure  2). 
Table  1 shows  part  of  the  results  of  K-sequence  in  structured  and  random  forms.  For  the  structured  form,  one 
maximum  18-sequence  was  found  (S  = <217,  210,  207,  200,  221,  225,  236,  240,  257,  248,  267,  272,  277,  282, 
306,  301,  292,  287>),  followed  by  two  17-sequences,  three  16-sequences,  and  so  on.  For  the  random  form, 
only  2-sequence  (S  = <64,  65>)  to  4-sequence  (S  = <142,  163,  204,  220>)  were  found. 


Figure  1 . A sequential  pattern  for  a structured  form 


Figure  2.  A sequential  pattern  for  a random  form 


From  the  sequence  length  results,  we  conclude  that  the  stimulus  layouts  may  have  implicit  visual  guiding 
effects.  In  the  random  form,  the  individual  showed  very  inconsistent  scanning  patterns  from  each  other,  and  a 
change  was  found  in  the  orientations  for  cancelling  the  next  target  without  following  any  horizontal  or 
vertical  direction.  Woodword  (1972)  indicated  how  the  short  and  vertical  scanning  pattern  was  influenced  by 
each  individual's  searching  orientation  and  majorly  by  the  next  possible  stimulus  one  attended  around.  The 
sequence  results  found  in  this  study  supports  the  layout  effects  for  visual  scanning. 
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Table  1.  Comparisons  of  the  number  of  K-sequence  on  the  two  stimulus  layout 


Structured  form 

Random  form 

K-sequence 

No.  of  K-sequence 

K-sequence 

No.  of  K-sequence 

18 

1 

- 

- 

17 

2 

- 

- 

16 

3 

- 

- 

4 

35 

4 

1 

3 

46 

3 

11 

2 

84 

2 

75 

4.  CONCLUSION 

In  this  paper,  a visualized  sequential  pattern  mining  approach  to  visuospatial  attention  has  been  developed. 
The  results  of  the  preliminary  experiment  also  revealed  that  the  stimulus  layout  would  affect  the  sequential 
patterns  of  visual  scanning  in  attention  process,  which  causes  significant  discrepancies  in  sequential  patterns 
between  the  searching  behavior  in  structured  and  random  arrays.  Further  issues  for  the  relationship  between 
the  patterns  of  visual  scanning  and  the  performance  of  visuospatial  attention  will  be  explored. 
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ABSTRACT 

The  purpose  of  the  paper  is  to  examine  whether  the  sub-constructs  of  self-determination,  that  is,  learners’  perceived  level 
of  autonomy,  competence,  and  relatedness,  predict  learning  flow,  persistence,  and  achievement  in  a blended  learning 
context.  Participants  are  102  adult  learners  who  voluntarily  registered  for  a Chinese  language  learning  seminar,  which 
consisted  of  weekly  offline  meetings  and  supplementary  online  activities  for  24  weeks.  A series  of  regression  analysis 
revealed  that  autonomy,  competence,  and  relatedness  significantly  predicted  learning  flow,  which  significantly  predicted 
persistence.  Also,  competence  and  relatedness,  and  learning  flow  predicted  achievement. 


KEYWORDS 

Self-determination;  learning  flow;  persistence;  blended  learning 


1.  INTRODUCTION 

Adult  learners  initiate  and  maintain  behaviors  to  the  extent  that  they  believe  the  behaviors  will  result  in 
desired  outeomes.  Self-determination  theory,  according  to  Deci  and  Ryan  (1980,  2000),  is  related  to  this 
human  motivation,  concerned  with  supporting  intrinsic  tendencies  to  behave  in  effective  way.  Self- 
determination  addresses  three  psychological  needs:  autonomy,  competence,  and  relatedness.  Autonomy  is  a 
tendency  toward  self-organization  and  self-regulation.  Without  autonomy,  learners  may  experience  failure 
since  they  hardly  recognize  what  they  want  or  should  do.  Competence  is  a psychological  construct  seeking  to 
control  the  outcome  and  experience  mastery.  It  is  important  because  it  is  necessary  for  the  enjoyment  of  the 
activity.  While  autonomy  and  competence  are  essential  for  intrinsic  motivation  and  interest,  relatedness  adds 
another  value  to  the  perspective  of  motivation.  Relatedness  means  a desire  to  interact  and  be  connected  to 
others.  All  these  sub-constructs  of  self-determination  are  influential  to  learning  process  as  well  as  outcome, 
according  to  previous  studies  (e.g.  Deci  & Vansteenkiste,  2004;  Ryan,  Kuhl,  Deci,  1997).  Especially  when 
online  learning  component  is  added  on,  learners  are  given  more  flexibility  in  organizing  learning  path,  scope 
and  interaction,  which  might  increase  the  importance  of  self-determination. 

Based  on  the  literature  review,  this  study  selected  learning  flow,  persistence,  and  perceived  level  of 
achievement  as  outcome  variables,  since  flow  and  persistence  are  theoretically  relevant  to  self-determination. 
Learning  flow  has  been  reported  as  a construct  that  predicts  the  effectiveness  of  learning.  According  to 
Csikszentmihalyi  (1996),  learning  flow  is  the  optimal  experience  of  a mental  concentration.  When  a task  is 
demanding  enough  to  be  interesting,  but  not  too  difficult  to  cause  frustration,  it  offers  the  possibility  for  a 
positive  experience  of  being  fully  engaged  in  an  activity.  Persistence  is  intent-to-persist,  and  has  been 
considered  a noteworthy  variable  for  evaluation  of  success  in  learning  (Martinez,  2003).  Prior  studies 
reported  that  autonomy  and  relatedness  affects  flow  (Evelein,  Korthagen,  & Brekelmans,  2008;  Fu,  Su,  & 
Yu,  2009),  while  the  relationship  between  competence  and  flow  has  not  been  consistent  (Zhao  et  ah,  2011). 
Also,  self-determination  consistently  affected  persistence  and  achievement. 

However,  most  of  the  previous  studies  were  conducted  in  K-12  or  formal  education  settings.  Hence,  the 
researchers  aimed  to  investigate  the  effects  of  self-determination  on  flow,  persistence,  and  achievement  of 
adult  learners  who  voluntarily  participate  in  and  self-regulate  their  learning  process  and  outcome  in  a blended 
learning  environment.  Research  questions  are  as  follows:  (RQ  1)  Does  perceived  level  of  self-determination 
(autonomy,  competence,  and  relatedness)  predict  learning  flow?  (RQ  2)  Does  perceived  level  of 
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self-determination  (autonomy,  eompetenee,  and  relatedness)  and  learning  flow  prediet  learner  persistenee? 
(RQ  3)  Does  pereeived  level  of  self-determination  (autonomy,  eompetenee,  and  relatedness)  and  learning 
flow  prediet  learners’  pereeived  level  of  aehievement? 


2.  METHODOLOGY 

Partieipants  are  102  adult  learners  who  voluntarily  registered  for  a 24  week-long  Chinese  language  learning 
seminar,  eonsisting  of  weekly  offline  meetings  and  supplementary  online  aetivities.  The  topie  of  the  seminar 
is  Chinese  grammar,  voeabulary,  and  speaking,  while  the  online  eommunity  provided  resourees,  q & a,  and 
diseussions  among  learners.  Initially  there  were  106  learners,  but  4 of  them  provided  ineomplete  responses. 

All  of  the  researeh  variables  were  measured  by  survey  questionnaires.  A Work-related  Basie  Need 
Satisfaetion  Seale  developed  by  Broeek,  Vansteenkiste,  Witte,  Soenens  and  Lens  (2010)  was  used  to  measure 
the  self-determination  variables.  The  instrument  had  7,  6,  and  10  items  for  eaeh  sub-eategory,  and  the 
Cronbaeh’s  a from  the  study  data  were  .65,  .70,  and  .88,  respeetively.  The  Short  Flow  State  Seale  developed 
by  Martin  and  Jaekson  (2008)  was  used,  and  the  seale  had  9 items.  Cronbaeh’s  a from  the  study  data  was  .77. 
For  learning  persistenee,  6 item  instrument  developed  by  Shin  (2003)  was  used,  with  the  Cronbaeh’s  a of  .87. 
Lastly,  pereeived  level  of  aehievement  was  measured  using  the  instrument  by  Korean  National  Institute  for 
Lifelong  Learning  (2010).  13  items  were  used  and  Cronbaeh’s  a from  the  study  data  was  .93.  The  survey 
questionnaires  were  reviewed  and  validated  by  aeademie  and  field  experts. 

A series  of  multiple  regressions  was  used  for  the  analysis. 


3.  RESULTS 


Deseriptive  analysis  indieated  there  were  signifieant  eorrelations  among  all  the  variables  at  the  signifieanee 
level  of  .05,  and  the  eorrelation  eoeffieients  ranged  from  .23  to  .71.  Also,  the  assumptions  of  normal 
distribution  and  multieollinearity  were  met.  Table  1 presents  the  regression  analysis  results. 

Table  1.  Multiple  regression  analysis  results 


(n=102) 


Criterion 

variables 

Predicting  

variables 

Unstandardized  Standardized 

B Std.  Error  P 

t 

P 

VIF 

Autonomy 

.25 

.08 

.23 

2.98* 

.00 

1.31 

Learning 

Competence 

.36 

.06 

.49 

6.24* 

.00 

1.35 

Flow 

Relatedness 

.14 

.06 

.19 

2.37* 

.02 

1.34 

(RQl) 

(Constant) 

1.03 

.28 

3.64 

.00 

R"(adj.  R^)=.55(.53),  F= 

=39.12,  p=.00 

Autonomy 

.22 

.12 

.18 

1.85 

.07 

1.43 

Competence 

.03 

.09 

.03 

.26 

.79 

1.89 

Persistence 

Relatedness 

-.06 

.08 

-.07 

-.13 

.A1 

1.42 

(RQ2) 

Learning  flow 

.63 

.14 

.54 

4.57* 

.00 

2.20 

(Constant) 

1.17 

.41 

2.87 

.01 

R"(adj.  R^)=.40(.37),  F= 

=15.82,  p=.00 

Autonomy 

.05 

.09 

.04 

.54 

.59 

1.43 

Competence 

.19 

.08 

.21 

2.50* 

.01 

1.89 

Achievement 

Relatedness 

.27 

.07 

.31 

4.14* 

.00 

1.42 

(RQ3) 

Learning  flow 

.47 

.11 

.40 

4.31* 

.00 

2.20 

(Constant) 

.17 

.33 

.51 

.61 

RTadj.  R")=.63(.61),  F=40.38,  p=.00 

* p<  .05 
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As  illustrated  above,  regression  analysis  results  indicated  that  autonomy,  competence,  and  relatedness 
significantly  predicted  learning  flow  with  of  .55,  regarding  the  research  question  1.  On  the  other  hand, 
only  the  learning  flow  was  a significant  predictor  for  persistence  with  of  .40.  Lastly,  competence 
relatedness,  and  learning  flow  predicted  the  perceived  level  of  achievement  with  R^  of  .63. 


4.  CONCLUSION 

The  researchers  aimed  to  apply  self-determination  theory  to  adult  learners  who  participated  in  a language 
learning  seminar  program  delivered  in  a blended  learning  mode.  Especially,  competence  and  relatedness 
predicted  both  flow  and  perceived  level  of  achievement.  However,  all  of  the  self-determination  variables 
failed  to  directly  predict  persistence,  partly  because  of  the  research  context,  that  is,  the  characteristics  of 
seminar  program  and  participants.  During  the  poster  presentation,  detailed  descriptions  on  the  research 
context  and  blended  learning  intervention  will  be  provided  in  order  to  draw  meaningful  interpretation  and 
implications  from  the  results.  Suggestions  for  further  study  will  also  be  discussed. 
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